[openstack-dev] [kolla][nova] Safe guest shutdowns with kolla?

[Clint Byrum]

Greetings! We've been deploying with Kolla on CentOS 7 now for a while, and
we've recently noticed a rather troubling behavior when we shutdown
hypervisors.

Somewhere between systemd and libvirt's systemd-machined integration,
we see that guests get killed aggressively by SIGTERM'ing all of the
qemu-kvm processes. This seems to happen because they are scoped into
machine.slice, but systemd-machined is killed which drops those scopes
and thus results in killing off the machines.

In the past, we've used the libvirt-guests service when our libvirt was
running outside of containers. This worked splendidly, as we could
have it wait 5 minutes for VMs to attempt a graceful shutdown, avoiding
interrupting any running processes. But this service isn't available on
the host OS, as it won't be able to talk to libvirt inside the container.

The solution I've come up with for now is this:

[Unit]
Description=Manage libvirt guests in kolla safely
After=docker.service systemd-machined.service
Requires=docker.service

[Install]
WantedBy=sysinit.target

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutStopSec=400
ExecStart=/usr/bin/docker exec nova_libvirt /usr/libexec/libvirt-guests.sh start
ExecStart=/usr/bin/docker start nova_compute
ExecStop=/usr/bin/docker stop nova_compute
ExecStop=/usr/bin/docker exec nova_libvirt /usr/libexec/libvirt-guests.sh 
shutdown

This doesn't seem to work, though I'm still trying to work out
the ordering and such. It should ensure that before we stop the
systemd-machined and destroy all of its scopes (thus, killing all the
vms), we run the libvirt-guests.sh script to try and shut them down. The
TimeoutStopSec=400 is because the script itself waits 300 seconds for any
VM that refuses to shutdown cleanly, so this gives it a chance to wait
for at least one of those. This is an imperfect solution but it allows us
to move forward after having made a reasonable attempt at clean shutdowns.

Anyway, just wondering if anybody else using kolla-ansible or kolla
containers in general have run into this problem, and whether or not
there are better/known solutions.

Thanks!

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][all] CD tangent - was: A culture change (nitpicking)

[Clint Byrum]

Quoting Sean McGinnis (2018-05-31 09:54:46)
> On 05/31/2018 03:50 AM, Thierry Carrez wrote:
> > Right... There might be a reasonable middle ground between "every 
> > commit on master must be backward-compatible" and "rip out all 
> > testing" that allows us to routinely revert broken feature commits (as 
> > long as they don't cross a release boundary).
> >
> > To be fair, I'm pretty sure that's already the case: we did revert 
> > feature commits on master in the past, therefore breaking backward 
> > compatibility if someone started to use that feature right away. It's 
> > the issue with implicit rules: everyone interprets them the way they 
> > want... So I think that could use some explicit clarification.
> >
> > [ This tangent should probably gets its own thread to not disrupt the 
> > no-nitpicking discussion ]
> >
> Just one last one on this, then I'm hoping this tangent ends.
> 
> I think what Thierry said is exactly what Dims and I were saying. I'm 
> not sure how that turned into
> the idea of supporting committing broken code. The point (at least mine) 
> was just that we should
> not have the mindset that HEAD~4 committed something that we realize was 
> not right, so we
> should not have the mindset that "someone might have deployed that 
> broken behavior so we
> need to make sure we don't break them." HEAD should always be 
> deployable, just not treated like
> an official release that needs to be maintained.
> 

We are what we test.

We don't test upgrading from one commit to the next. We test upgrading
from the previous stable release. And as such, that's what has to keep
working.

So no, a revert shouldn't ever be subject to "oh no somebody may have
deployed this and you don't revert the db change". That's definitely a
downstream consideration and those who CD things have ways of detecting
and dealing with this on their end. That said, it would be nice if
developers consider this corner case, and try not to make it a huge
mess to unwind.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo] storyboard evaluation

[Clint Byrum]

Excerpts from Thierry Carrez's message of 2018-01-17 11:51:52 +0100:
> Emilien Macchi wrote:
> > On Tue, Jan 16, 2018 at 8:29 AM, Jeremy Stanley  wrote:
> >>> - how do we deal milestones in stories and also how can we have a
> >>> dashboard with an overview per milestone (useful for PTL + TripleO
> >>> release managers).
> >>
> >> So far, the general suggestion for stuff like this is to settle on a
> >> consistent set of story tags to apply. It really depends on whether
> >> you're trying to track this at a story or task level (there is no
> >> per-task tagging implemented yet at any rate). I could imagine, for
> >> example, setting something like tripleo-r2 as a tag on stories whose
> >> TripleO deliverable tasks are targeting Rocky milestone #2, and then
> >> you could have an automatic board with stories matching that tag and
> >> lanes based on the story status.
> > 
> > Does this kind of board exist already?
> 
> Rather than using tags, you can make a Board itself your "milestone
> view". To make a task/story part of the milestone objectives, you just
> add it to your board. Then use various lanes on that board to track
> progress.
> 
> See the "Zuul v3 Operational" board in
> https://storyboard-blog.sotk.co.uk/things-that-storyboard-does-differently.html
> for an example -- I think it's pretty close to what you need.
> 
> I /think/ if you used a tag you'd miss a feature: the ability to specify
> a board lane as needing to automatically contain "all things that match
> a given criteria (like a tag match) but which would not already appear
> in one of the other lanes on this board". *And* allow to move things
> from that automatic lane to the other lanes. That way you can have a
> board that automatically contains all the things that match your tag (by
> default in the automatic lane), but still lets you move things around
> onto various lanes.
> 
> I don't think that exists, which is why I'd use a Board directly as a
> "milestone tracker", rather than go through tagging.
> 

That particular example board was built from tasks semi-automatically,
using a tag, by this script running on a cron job somewhere:

https://git.openstack.org/cgit/openstack-infra/zuul/tree/tools/update-storyboard.py?h=feature/zuulv3

We did this so that we could have a rule "any task that is open with
the zuulv3 tag must be on this board". Jim very astutely noticed that
I was not very good at being a robot that did this and thus created the
script to ease me into retirement from zuul project management.

The script adds new things in New, and moves tasks automatically to
In Progress, and then removes them when they are completed. We would
periodically groom the "New" items into an appropriate lane with the hopes
of building what you might call a rolling-sprint in Todo, and calling
out blocked tasks in a regular meeting. Stories were added manually as
a way to say "look in here and add tasks", and manually removed when
the larger effort of the story was considered done.

I rather like the semi-automatic nature of it, and would definitely
suggest that something like this be included in Storyboard if other
groups find the board building script useful. This made a cross-project
effort between Nodepool and Zuul go more smoothly as we had some more
casual contributors to both, and some more full-time.

The best part about a tag is that it is readily visible on the story
view. I think given that alone, tags are a pretty valid way to call out
what milestone you think a story is aimed at.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching to longer development cycles

[Clint Byrum]

Excerpts from Thomas Goirand's message of 2017-12-15 16:15:04 +0100:
> On 12/14/2017 12:44 AM, Clint Byrum wrote:
> > We can take stock of the intermediate releases over the last year, and make
> > sure they all work together once a year. Chris Jones mentioned that we 
> > should
> > give users time between milestones and release. I suggest we release an
> > intermediary and _support it_. Let distros pick those up when they need to 
> > ship
> > new features.
> 
> I don't think this will happen.
> 
> > Let users jump ahead for a few projects when they need the bug fixes.
> 
> And that, I don't agree. New releases aren't to fix bugs, bugs should be
> fixed in stable too, otherwise you face new issues trying to get a
> bugfix. And that's why we have stable.
> 

We have stable for critical bug fixes. However, performance enhancements,
scalability improvements, API weirdness, are not backported. In general,
most of what arrives in each commit is good for most users. Bug fixes
are not all backported. If a regression happens, it happens because it
leverages a scenario we don't properly test.

> > I understand the belief that nobody will run the intermediaries.
> 
> Not only that. Everyone is lagging a few release behind, and currently,
> upstream OpenStack don't care backporting to older releases.
> 

Right but if you de-couple projects a bit, folks can upgrade the stuff that
they need to when they need to. As Matt R. says, this kinda works today. I
suggest we start testing it more and asserting that it works.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching to longer development cycles

[Clint Byrum]

I feel like the thing not stated is that what we really don't like is
that our users are falling behind. Sure, pressure in release time is
definitely real. Multiple PTG's and forums feels like people are
missing out. But what's really awful is that some of our users are
stuck down in the bottom of an upgrade well and the only supported
path out is through several upgrades.

What if we did a community goal to get upgrade testing solidified,
and have it test upgrading each project from _three_ stable releases
prior. So, if we did this for Rocky, we'd make sure you could upgrade
to Rocky from Queens, Pike, or Ocata.

And then we keep doing that in perpetuity. This gives users 18 months
from one release to get it up and stabilized, and then start the upgrade
cycle again. It also gives them 2 targets to hit while they do that,
since they can try "the stable that just released" and "the prior one"
and not feel like they'll fall behind if the new release has issues and
they have to stay conservative.

Basically instead of doing the release less frequently, shift the value
of the release toward what our users need.

Excerpts from Thierry Carrez's message of 2017-12-13 17:17:26 +0100:
> Hi everyone,
> 
> Over the past year, it has become pretty obvious to me that our
> self-imposed rhythm no longer matches our natural pace. It feels like we
> are always running elections, feature freeze is always just around the
> corner, we lose too much time to events, and generally the impression
> that there is less time to get things done. Milestones in the
> development cycles are mostly useless now as they fly past us too fast.
> A lot of other people reported that same feeling.
> 
> As our various components mature, we have less quick-paced feature
> development going on. As more and more people adopt OpenStack, we are
> more careful about not breaking them, which takes additional time. The
> end result is that getting anything done in OpenStack takes more time
> than it used to, but we have kept our cycle rhythm mostly the same.
> 
> Our development pace was also designed for fast development in a time
> where most contributors were full time on OpenStack. But fewer and fewer
> people have 100% of their time to dedicate to OpenStack upstream
> development: a lot of us now have composite jobs or have to participate
> in multiple communities. This is a good thing, and it will only
> accelerate as more and more OpenStack development becomes fueled
> directly by OpenStack operators and users.
> 
> In another thread, John Dickinson suggested that we move to one-year
> development cycles, and I've been thinking a lot about it. I now think
> it is actually the right way to reconcile our self-imposed rhythm with
> the current pace of development, and I would like us to consider
> switching to year-long development cycles for coordinated releases as
> soon as possible.
> 
> What it means:
> 
> - We'd only do one *coordinated* release of the OpenStack components per
> year, and maintain one stable branch per year
> - We'd elect PTLs for one year instead of every 6 months
> - We'd only have one set of community goals per year
> - We'd have only one PTG with all teams each year
> 
> What it does _not_ mean:
> 
> - It doesn't mean we'd release components less early or less often. Any
> project that is in feature development or wants to ship changes more
> often is encouraged to use the cycle-with-intermediary release model and
> release very early and very often. It just means that the minimum we'd
> impose for mature components is one release per year instead of one
> release every 6 months.
> 
> - It doesn't mean that we encourage slowing down and procrastination.
> Each project would be able to set its own pace. We'd actually encourage
> teams to set objectives for the various (now longer) milestones in the
> cycle, and organize virtual sprints to get specific objectives done as a
> group. Slowing down the time will likely let us do a better job at
> organizing the work that is happening within a cycle.
> 
> - It doesn't mean that teams can only meet in-person once a year.
> Summits would still provide a venue for team members to have an
> in-person meeting. I also expect a revival of the team-organized
> midcycles to replace the second PTG for teams that need or want to meet
> more often.
> 
> - It doesn't mean less emphasis on common goals. While we'd set goals
> only once per year, I hope that having one full year to complete those
> will let us tackle more ambitious goals, or more of them in parallel.
> 
> - It doesn't simplify upgrades. The main issue with the pace of
> upgrading is not the rhythm, it's the imposed timing. Being forced to
> upgrade every year is only incrementally better than being forced to
> upgrade every 6 months. The real solution there is better support for
> skipping releases that don't matter to you, not longer development cycles.
> 
> - It doesn't give us LTS. The cost of maintaining branches is not rea

Re: [openstack-dev] [all] Switching to longer development cycles

[Clint Byrum]

Excerpts from Ed Leafe's message of 2017-12-13 22:55:43 -0600:
> On Dec 13, 2017, at 4:38 PM, German Eichberger 
>  wrote:
> 
> > It looks like the implicit expectation is that devs also need to attend the 
> > Forums at the summit in addition to the PTG. The Forums, though important, 
> > hardly made it worthwhile for me to attend the summit (and in fact I 
> > skipped Sydney). On the other hand some devs got together and hashed out 
> > some plans for their projects. Personally, I feel the PTG is not working if 
> > we also have summits – and having two summits and one PTG will make things 
> > worse. Therefore I propose to scrap the PTG and add “design summits” back 
> > to the OpenStack summit. As a side effect this will be a better on-ramp for 
> > casual developers who can’t justify going to the PTG and ensure enough 
> > developers are on-hand to hear the operator’s feedback.
> 
> The original purpose of the summits were for the developers to get together 
> to plan what they would work on for the next few months. Over time, the big 
> money came pouring in, and they became huge marketing events, with developers 
> pushed off to the fringes. The recent PTGs have been refreshing because they 
> are more like the original events, where there is no distraction by the 
> corporate marketing machines, and we can focus on getting shit done.
> 
> My question is: if we only have a release once a year, why do we need two 
> Summits a year?
> 

Regional rotation seems the best reason to keep two per year.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching to longer development cycles

[Clint Byrum]

I think this is in-line with my earlier points. Projects started later,
with clear API definitions, are easier to run on whatever version you
need. They don't dig in under the hood the way Neutron/Nova/Cinder do
because of their shared history.

It also helps that they don't require a bunch of patches out-of-tree to
scale. Luckily, I'm told neither does Nova anymore, so maybe this will
get more natural.. well.. naturally.

Excerpts from Blair Bethwaite's message of 2017-12-14 18:39:33 +1100:
> The former - we're running Cells so only have a single region currently
> (except for Swift where we have multiple proxy endpoints around the
> country, all backed by a global cluster, but they have to be different
> regions to put them all in the service catalog). See
> https://trello.com/b/9fkuT1eU/nectar-openstack-versions for the current
> version snapshot.
> 
> On 14 Dec. 2017 18:00, "Clint Byrum"  wrote:
> 
> > Excerpts from Blair Bethwaite's message of 2017-12-14 17:44:53 +1100:
> > > On 14 December 2017 at 17:36, Clint Byrum  wrote:
> > > > The batch size for "upgrade the whole cloud" is too big. Let's help our
> > > > users advance components one at a time, and then we won't have to worry
> > > > so much about doing the whole integrated release dance so often.
> > >
> > > Is there any data about how operators approach this currently? Nectar
> > > (and I presume other large and/or loosely coordinated OpenStack
> > > clouds) has been running different projects across multiple versions
> > > for quite a while, sometimes 3 or 4 different versions. Coordinating
> > > upgrades in a federated cloud with distributed operations requires
> > > that we do this, e.g., our current Nova Newton upgrade has probably
> > > been in-train for a couple of months now.
> > >
> >
> > That's interesting. Can you share what you mean by running 3 or 4
> > different versions?
> >
> > Do you mean you mix versions in a single region, like, Pike keystone,
> > Ocata Nova, and Newton Neutron? Or do you mean you might have a region
> > running Pike, and another running Ocata, and another running Newton?
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching to longer development cycles

[Clint Byrum]

Excerpts from Blair Bethwaite's message of 2017-12-14 17:44:53 +1100:
> On 14 December 2017 at 17:36, Clint Byrum  wrote:
> > The batch size for "upgrade the whole cloud" is too big. Let's help our
> > users advance components one at a time, and then we won't have to worry
> > so much about doing the whole integrated release dance so often.
> 
> Is there any data about how operators approach this currently? Nectar
> (and I presume other large and/or loosely coordinated OpenStack
> clouds) has been running different projects across multiple versions
> for quite a while, sometimes 3 or 4 different versions. Coordinating
> upgrades in a federated cloud with distributed operations requires
> that we do this, e.g., our current Nova Newton upgrade has probably
> been in-train for a couple of months now.
> 

That's interesting. Can you share what you mean by running 3 or 4
different versions?

Do you mean you mix versions in a single region, like, Pike keystone,
Ocata Nova, and Newton Neutron? Or do you mean you might have a region
running Pike, and another running Ocata, and another running Newton?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching to longer development cycles

[Clint Byrum]

Excerpts from Ed Leafe's message of 2017-12-13 23:02:11 -0600:
> On Dec 13, 2017, at 5:44 PM, Clint Byrum  wrote:
> 
> > One thing I've always admired about Swift was how immune to the cadence 
> > Swift
> > appears to be.
> 
> As I've pointed out before [0], Swift is a whole 'nother thing.
> 
> It does not have API interdependencies with anything else in OpenStack. It is 
> free to do things its own way on its own schedule.

_this is exactly my point_

First, Swift has plenty of API interdependency. Swift consumes Keystone's
API just like everyone else. Swift also fronts a glance driver which is
optional to use. Trove backs things up to Swift.

The reason it doesn't feel like they're entangled with the rest of OpenStack
_is because they aren't_.

> 
> The rest of OpenStack is what Nova originally was. It was split into many 
> different project because of the sheer complexity of the tasks it had to 
> perform. But splitting it up required that we occasionally have to make sure 
> that we're all still working together well.
> 

My entire point is that we should draw clearer lines around our API's and
admit that we have a ton of tight coupling that requires us to release
things in an integrated fashion.

While I agree, where we're at is a tightly coupled mess, what I don't agree
with is that the answer is to keep shipping the mess.

I've said this before and honestly, I got crickets back in Atlanta for
saying it, but nova-compute should be renamed 'hypervisor'. Anything that
needs to run on a hypervisor should be in this service's world, and not
in the respective projects. Each hypervisor host is an endpoint of this
API, placement being the service discovery mechanism for that endpoint.

Neutron and Cinder would be treated as first-class API consumers of
the hypervisor API. If you want to upgrade cinder, go ahead, as long as
the hypervisor API version you have is supported by Cinder, Cinder can
control the volumes on it.

Right now we just can't do that. We know they were split out and we let
them share a filesystem and a network namespace on nova-compute and keep
a tight ratchet on that poorly defined API by releasing everything from
neutron agents to brick versions in lock-step. As long as we do that,
we're stuck with a massive monolith of a product to ship, and that is
likely not serving our users.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching to longer development cycles

[Clint Byrum]

Excerpts from Ed Leafe's message of 2017-12-13 22:51:19 -0600:
> On Dec 13, 2017, at 4:31 PM, Doug Hellmann  wrote:
> 
> > You're missing the key point that coupled with the change in the
> > overall development cycle schedule we would be encouraging projects
> > to release more often.  I'd personally rather do away with milestones
> > altogether and just tag everything monthly, but some projects may
> > not be ready to do that and some may want to go more often (libraries
> > in particular).
> 
> I think you're missing the reality that intermediate releases have about zero 
> uptake in the real world. We have had milestone releases of Nova for years, 
> but I challenge you to find me one non-trivial deployment that uses one of 
> them. To my knowledge, based on user surveys, it is only the major 6-month 
> named releases that are deployed, and even then, some time after their 
> release.
> 
> Integrated releases make sense for deployers. What does it mean if Nova has 
> some new stuff, but it requires a new release from Cinder in order to use it, 
> and Cinder hasn't yet released the necessary updates? Talking about releasing 
> projects on a monthly-tagged basis just dumps the problem of determining what 
> works with the rest of the codebase onto the deployers.
> 
> 

This isn't really fair to the users. They're running what we tell them
to run. They're also running what we try to test together, which is all
the same versions of the same components at integrated release time.

While I do think most users _should_ stick to what's tested together,
and we should keep testing things together, I also think we should be
more comfortable telling users to go ahead and install a new release of
Nova and feel confident they're going to be supported in doing so.

The batch size for "upgrade the whole cloud" is too big. Let's help our
users advance components one at a time, and then we won't have to worry
so much about doing the whole integrated release dance so often.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Switching to longer development cycles

[Clint Byrum]

Excerpts from Thierry Carrez's message of 2017-12-13 17:17:26 +0100:
> Hi everyone,
> 
> Over the past year, it has become pretty obvious to me that our
> self-imposed rhythm no longer matches our natural pace. It feels like we
> are always running elections, feature freeze is always just around the
> corner, we lose too much time to events, and generally the impression
> that there is less time to get things done. Milestones in the
> development cycles are mostly useless now as they fly past us too fast.
> A lot of other people reported that same feeling.
> 
> As our various components mature, we have less quick-paced feature
> development going on. As more and more people adopt OpenStack, we are
> more careful about not breaking them, which takes additional time. The
> end result is that getting anything done in OpenStack takes more time
> than it used to, but we have kept our cycle rhythm mostly the same.
> 
> Our development pace was also designed for fast development in a time
> where most contributors were full time on OpenStack. But fewer and fewer
> people have 100% of their time to dedicate to OpenStack upstream
> development: a lot of us now have composite jobs or have to participate
> in multiple communities. This is a good thing, and it will only
> accelerate as more and more OpenStack development becomes fueled
> directly by OpenStack operators and users.
> 
> In another thread, John Dickinson suggested that we move to one-year
> development cycles, and I've been thinking a lot about it. I now think
> it is actually the right way to reconcile our self-imposed rhythm with
> the current pace of development, and I would like us to consider
> switching to year-long development cycles for coordinated releases as
> soon as possible.
> 
> What it means:
> 
> - We'd only do one *coordinated* release of the OpenStack components per
> year, and maintain one stable branch per year


So I want to zero in on this part of the proposal.

It feels like the major driving factor for frustration with the 6 month cadence
is the weight of the coordinated release versus the value, both for developers
and users.

But, I wonder, why do we put so much emphasis on this one process?

One thing I've always admired about Swift was how immune to the cadence Swift
appears to be.

If a set of enthusiastic developers shows up mid-cycle and starts hammering on
a new feature, there's no angst about whether or not it can be done "this
release". It's worked on, Swift moves forward, and when it is done, Swift
releases.

We have this amazing tool that lets us get away with not doing the hard work
Swift did to encapsulate itself. Zuul has enabled us to test all the projects
that want to co-gate together as one. This allows co-dependencies to creep in
under the covers. We have something that, on first look, appears to be
micro-service-architecture, but then in reality, behaves like a massive
monolithic piece of software.

But maybe we're papering over the real problem. Maybe we need to stop working
so hard to coordinate these releases.

As projects mature, I'd expect some to slow down and some to speed up. Glance,
for instance, is in perpetual maintenance mode. Nova is currently in a pretty
heavy debt pay down state and still landing features. Neutron too. So I'd
expect Glance to make frequent small releases of newly fixed items. Nova might
need to do a slower release while a refactor happens, and then a few follow-ups
to get fixes and small enhancements out.

I understand you mentioned the intermediate release tag and practice. I'm
suggesting that we should focus on getting _everyone_ on that train. I don't
think this works without such an effort.

We can keep the release name cadence. But then we get the best of both worlds.
We can take stock of the intermediate releases over the last year, and make
sure they all work together once a year. Chris Jones mentioned that we should
give users time between milestones and release. I suggest we release an
intermediary and _support it_. Let distros pick those up when they need to ship
new features. Let users jump ahead for a few projects when they need the bug
fixes. And then once a year, have a flag day where we all stop, and make an
assertion that we've done some things, achieved some goals, and suggest that
you upgrade all the components to at least that version, and that we'll make
sure you can do that from the last big version.

I understand the belief that nobody will run the intermediaries. If we don't
make them work independently of other projects, nobody will. But if we do, and
if we make sure our interfaces are solid enough to advance components at a
disjoint pace, then users will in fact start to do just that, and we'll retain
the faster feedback cycle, without the heavy community load of coordinated
releases every 6 months.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstac

Re: [openstack-dev] [all] Switching to longer development cycles

[Clint Byrum]

Excerpts from Chris Jones's message of 2017-12-13 19:25:03 +:
> Hey
> 
> On 13 December 2017 at 17:31, Thierry Carrez  wrote:
> 
> > See attached for the PDF strawman the release team came up with when
> > considering how that change would roll out in practice...
> >
> 
> [in which the final stage of the release is 8 weeks, therefore shorter than
> the other 10/11 week stages]
> 
> I'm not going to go on about this beyond this mail, since I've been roundly
> shot down about this before, but please could we have the gap between the
> final milestone and the release, be longer? Every extra week there is more
> release quality :)
> 

I want to believe that is true.

However I have no real evidence that it is.

What class of users runs RC's in OpenStack?

With narrower scoped projects, running RC's is pretty common. If you're a PHP
shop, you might have automated testing setup to test your code with the RC so
you can upgrade sooner. But you might not bother to run RC's of MySQL, because
you're happy with the feature set and comfortable with the amount of time the
release you're running is supported.

However, with OpenStack's "upgrade all the things" approach to coordinated
releases and testing, this becomes a much higher bar for running an RC.

My somewhat educated guess is that our automated CI is about the only thing
beyond a few developers testing their own new features that will run this code
before it is released.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc] [all] TC Report 49

[Clint Byrum]

Excerpts from Graham Hayes's message of 2017-12-07 17:08:55 +:
> 
> On 06/12/17 23:33, James E. Blair wrote:
> > Chris Dent  writes:
> > 
> >> The expansion of the Foundation was talked about at the summit in
> >> Sydney, but having something happen this quickly was a bit of a
> >> surprise, leading to some [questions in
> >> IRC](http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2017-12-05.log.html#t2017-12-05T14:11:33)
> >> today. Jonathan Bryce showed up to help answer them.
> > 
> > I'd like to address a misconception in that IRC log:
> > 
> > 2017-12-05T14:20:56   it does not take long to create a repo on 
> > our infrastructure
> > 2017-12-05T14:21:14   though I guess without the name 
> > flattening, it would have been an "openstack" repository
> > 
> > While there's still some work to be done on flattening the namespace for
> > existing repos, I think it would be quite straightforward to create a
> > repository for a non-openstack project in gerrit with no prefix (or, of
> > course, a different prefix).  I don't think that would have been an
> > obstacle.
> > 
> > And regarding this:
> > 
> > 2017-12-05T15:05:30   i'm not sure how much of infra's ci they 
> > could make use of given https://github.com/kata-containers/tests
> > 
> > I don't see an obstacle to using Zuul right now either -- even before
> > they have tests.
> 
> It is worth remembering that this is a completely separate project to
> OpenStack, with its own governance. They are free not to use our tooling
> (and considering a lot of containers work is on github, they may never
> use it).
> 

It's worth noting that while that precludes Gerrit, it does not preclude
Zuul, which has a GitHub driver, though we're still not sure if it scales
as well as Gerrit.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Removing internet access from unit test gates

[Clint Byrum]

Excerpts from Jens Harbott's message of 2017-11-24 13:22:23 +:
> 2017-11-21 15:04 GMT+00:00 Jeremy Stanley :
> > On 2017-11-21 09:28:20 +0100 (+0100), Thomas Goirand wrote:
> > [...]
> >> The only way that I see going forward, is having internet access
> >> removed from unit tests in the gate, or probably just the above
> >> variables set.
> > [...]
> ...
> > Removing network access from the machines running these jobs won't
> > work, of course, because our job scheduling and execution service
> > needs to reach them over the Internet to start jobs, monitor
> > progress and collect results.
> 
> I have tested a variant that would accomodate this: Run the tests in a
> new network namespace that no network configuration at all. There are
> some issues with this still:
> 
> - One needs sudo access in order to run something similar to "ip netns
> exec ns1 tox ...". This could still be set up in a way such that the
> tox user/environment itself does not need sudo.
> - I found some unit tests that do need to talk to localhost, so one
> still has to setup lo with 127.0.0.1/32.
> - Most important issue that prevents me from successfully running tox
> currently though is that even if I prepared the venv beforehand with
> "tox -epy27 --notest", the next tox run will still want to reinstall
> the project itself and most projects have something like
> 
> install_command =
> pip install -U
> -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt}
> {opts} {packages}
> 
> in their tox.ini, which will obviously fail without network
> connectivity. Running something like
> 
> sudo ip netns exec ns1 su -c ".tox/py27/bin/stestr run" $USER
> 
> does work rather well though. Does anyone have an idea how to force
> tox to just run the tests without doing any installation steps? Then I
> guess one could come up with a small wrapper to handle the other
> steps.
> 

Tox can be run without tests first to build all the venvs:

  $ tox --notest

Then with a sufficiently new kernel or setuid bwrap, one can use
Bubblewrap to get a clean netns:

bwrap --unshare-net 

Unfortunately bubblewrap is pretty new, so it's only going to be there
in package repos with newer Fedora and Ubuntu.

Either way, this is pretty doable with Zuulv3 inheritance. Whatever
jobs are using as parent, make a -nonet child of that which runs tox
--notest with network access still intact, then by whatever means makes
the most sense, namespaces or firewalls, runs the tests themselves with
access restricted.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] key_pair update on rebuild (a whole lot of conversations)

[Clint Byrum]

Excerpts from Ben Nemec's message of 2017-10-03 23:05:49 -0500:
> 
> On 10/03/2017 03:16 PM, Sean Dague wrote:
> > = Where I think we are? =
> > 
> > I think with all this data we're at the following:
> > 
> > Q: Should we add this to rebuild
> > A: Yes, probably - after some enhancement to the spec *
> > 
> > * - we really should have much better use cases about the situations it
> > is expected to be used in. We spend a lot of time 2 and 3 years out
> > trying to figure out how anyone would ever use a feature, and adding
> > another one without this doesn't seem good
> 
> Here's an example from my use: I create a Heat stack, then realize I 
> deployed some of the instances with the wrong keypair.  I'd rather not 
> tear down the entire stack just to fix that, and being able to change 
> keys on rebuild would allow me to avoid doing so.  I can rebuild a 
> Heat-owned instance without causing any trouble, but I can't re-create it.
> 
> I don't know how common this is, but it's definitely something that has 
> happened to me in the past.
> 

Sorry but this is an argument to use Heat more, but rebuild is totally
unnecessary.

In heat if you change the keypair and update the stack, it will create a new
one with the right keypair and delete the old instance (or you can make it use
rebuild, a feature I believe I developed actually). The updated IPs will be
rolled out to all resources that reference that instance's IP. If you have wait
conditions which depend on this instance, Heat will wait until they are
re-triggered before deleting the old instance. This is literally why Heat is a
cool thing, because it lets you use the cloud the way the cloud was intended to
be used.

If you use rebuild, while it is rebuilding, your service is unavailable. If you
use create/wait/delete you have a chance to automate the transition from the
old to new instance.

> > 
> > Q: should this also be on reboot?
> > A: NO - it would be too fragile
> > 
> > 
> > I also think figuring out a way to get Nova out of the key storage
> > business (which it really shouldn't be in) would be good. So if anyone
> > wants to tackle Nova using Barbican for keys, that would be ++. Rebuild
> > doesn't wait on that, but Barbican urls for keys seems like a much
> > better world to be in.
> > 
> > -Sean
> > 
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Fwd: [Openstack-operators][tc] [keystone][all] v2.0 API removal

[Clint Byrum]

Excerpts from Jeremy Stanley's message of 2017-10-21 13:37:01 +:
> On 2017-10-20 22:50:53 + (+), Fox, Kevin M wrote:
> [...]
> > Ideally, there should be an OpenStack overarching architecture
> > team of some sort to handle this kind of thing I think.
> 
> There was one for a while, but it dissolved due to lack of community
> participation. If you'd like to help reboot it, Clint B. can
> probably provide you with background on the previous attempt.
> 

I'd be in support of reviving the Architecture Working Group (SIG?).

Would need to see more people commit to it though. It mostly felt like
a place for Thierry and me to write down our ideas, and a title to put
on a room at the PTG so we could have cross-project discussions about
our ideas.

That said, there is a cross-project process that works pretty well when
one project needs to ask for changes from other projects:

https://docs.openstack.org/project-team-guide/cross-project.html

I believe the Keystone team followed this process despite some fumbles
early in the v3 story.

> > Without such an entity though, I think the TC is probably
> > currently the best place to discuss it though?
> 
> Contrary to the impression some people seem to have, the TC is not
> primarily composed of cloud architects; it's an elected body of
> community leaders who seek informed input from people like you. I've
> personally found no fault in the process and timeline the Keystone
> team followed in this situation but I'm also not the primary
> audience for their software, so it's good to hear from those who are
> about ways to improve similar cases in the future. However, I also
> understand that no matter how widely and carefully changes are
> communicated, there's only so much anyone can do to avoid surprising
> the subset of users who simply don't pay attention.

Right, the TC is more or less a legislative body. They can set policy
but they don't actually make sure the vision is implemented directly.

I made an argument that there's a need for an executive branch to get
hard things done here:

http://fewbar.com/2017/02/open-source-governance-needs-presidents/

Without some kind of immediate executive that sits above project levels,
we'll always be designing by committee and find our silos getting deeper.

All of that said, I believe the Keystone team did a great job of getting
something hard done. As Morgan states, it was a 100% necessary evolution
and required delicate orchestration. Well done Keystone team!

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Supporting SSH host certificates

[Clint Byrum]

And k8s has the benefit of already having been installed with certs that
had to get there somehow.. through a trust bootstrap.. usually SSH. ;)

Excerpts from Fox, Kevin M's message of 2017-10-09 17:37:17 +:
> Yeah, there is a way to do it today. it really sucks though for most users. 
> Due to the complexity of doing the task though, most users just have gotten 
> into the terrible habit of ignoring the "this host's ssh key changed" and 
> just blindly accepting the change. I kind of hate to say it this way, but 
> because of the way things are done today, OpenStack's training folks to 
> ignore man in the middle attacks. This is not good. We shouldn't just shrug 
> it off and say folks should be more careful. We should try and make the edge 
> less sharp so they are less likely to stab themselves, and later, give 
> OpenStack a bad name because OpenStack was involved.
> 

I agree that we could do better.

I think there _is_ a standardized method which is to print the host
public keys to console, and scrape them out on first access.

> (Yeah, I get it is not exactly OpenStack's fault that they use it in an 
> unsafe manner. But still, if OpenStack can do something about it, it would be 
> better for everyone involved)
> 

We could do better though. We could have an API for that.

> This is one thing I think k8s is doing really well. kubectl execuses 
> the chain of trust built up from user all the way to the pod. There isn't 
> anything manual the user has to do to secure the path. OpenStack really could 
> benefit from something similar for client to vm.
> 

This is an unfair comparison. k8s is running in the user space, and as
such rides on the bootstrap trust of whatever was used to install it.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Supporting SSH host certificates

[Clint Byrum]

Excerpts from Giuseppe de Candia's message of 2017-10-06 13:49:43 -0500:
> Hi Clint,
> 
> Isn't user-data by definition available via the Metadata API, which isn't
> considered secure:
> https://wiki.openstack.org/wiki/OSSN/OSSN-0074
> 

Correct! The thinking is to account for the MITM attack vector, not
host or instance security as a whole. One would hope the box comes up
in a mostly drone-like state until it can be hardened with a new secret
host key.

> Or is there a way to specify that certain user-data should only be
> available via config-drive (and not metadata api)?
> 
> Otherwise, the only difference I see compared to using Meta-data is that
> the process you describe is driven by the user vs. automated.
> 
> Regarding the extra plumbing, I'm not trying to avoid it. I'm thinking to
> eventually tie this all into Keystone. For example, a project should have
> Host CA and User CA keys. Let's assume OpenStack manages these for now,
> later we can consider OpenStack simply proxying signature requests and
> vouching that a public key does actually belong to a specific instance (and
> host-name) or Keystone user. So what I think should happen is when a
> Project is enabled for SSHaaS support, any VM instance automatically gets
> host certificate, authorized principal files based on Keystone roles for
> the project, and users can call an API (or Dashboard form) to get a public
> key signed (and assigned appropriate SSH principals).
> 

Fascinating, but it's hard for me to get excited about this when I can
just handle MITM security myself.

Note that the other existing techniques are simpler too. Most instances
will print the public host key to the console. The API offers console
access, so it can be scraped for the host key.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Supporting SSH host certificates

[Clint Byrum]

A long time ago, a few Canonical employees (Scott Moser was one of them,
forget who else was doing it, maybe Dave Walker and/or Dustin Kirkland)
worked out a scheme for general usage that doesn't require extra plumbing:

 * Client generates a small SSH host key locally and pushes it into
   user data in a way which causes the image to boot and install this
   key from user_data as the host SSH key.
 * Client SSH's in with the strict requirement that the host key be the
   one they just generated and pushed into the instance.
 * Client now triggers new host key generation, and copies new public
   key into client known_hosts.

With this system you don't have to scrape console logs for SSH keys or
build your system on hope.

Excerpts from Giuseppe de Candia's message of 2017-09-29 14:21:06 -0500:
> Hi Folks,
> 
> 
> 
> My intent in this e-mail is to solicit advice for how to inject SSH host
> certificates into VM instances, with minimal or no burden on users.
> 
> 
> 
> Background (skip if you're already familiar with SSH certificates): without
> host certificates, when clients ssh to a host for the first time (or after
> the host has been re-installed), they have to hope that there's no man in
> the middle and that the public key being presented actually belongs to the
> host they're trying to reach. The host's public key is stored in the
> client's known_hosts file. SSH host certicates eliminate the possibility of
> Man-in-the-Middle attack: a Certificate Authority public key is distributed
> to clients (and written to their known_hosts file with a special syntax and
> options); the host public key is signed by the CA, generating an SSH
> certificate that contains the hostname and validity period (among other
> things). When negotiating the ssh connection, the host presents its SSH
> host certificate and the client verifies that it was signed by the CA.
> 
> 
> 
> How to support SSH host certificates in OpenStack?
> 
> 
> 
> First, let's consider doing it by hand, instance by instance. The only
> solution I can think of is to VNC to the instance, copy the public key to
> my CA server, sign it, and then write the certificate back into the host
> (again via VNC). I cannot ssh without risking a MITM attack. What about
> using Nova user-data? User-data is exposed via the metadata service.
> Metadata is queried via http (reply transmitted in the clear, susceptible
> to snooping), and any compute node can query for any instance's
> meta-data/user-data.
> 
> 
> 
> At this point I have to admit I'm ignorant of details of cloud-init. I know
> cloud-init allows specifying SSH private keys (both for users and for SSH
> service). I have not yet studied how such information is securely injected
> into an instance. I assume it should only be made available via ConfigDrive
> rather than metadata-service (again, that service transmits in the clear).
> 
> 
> 
> What about providing SSH host certificates as a service in OpenStack? Let's
> keep out of scope issues around choosing and storing the CA keys, but the
> CA key is per project. What design supports setting up the SSH host
> certificate automatically for every VM instance?
> 
> 
> 
> I have looked at Vendor Data and I don't see a way to use that, mainly
> because 1) it doesn't take parameters, so you can't pass the public key
> out; and 2) it's queried over http, not https.
> 
> 
> 
> Just as a feasibility argument, one solution would be to modify Nova
> compute instance boot code. Nova compute can securely query a CA service
> asking for a triplet (private key, public key, SSH certificate) for the
> specific hostname. It can then inject the triplet using ConfigDrive. I
> believe this securely gets the private key into the instance.
> 
> 
> 
> I cannot figure out how to get the equivalent functionality without
> modifying Nova compute and the boot process. Every solution I can think of
> risks either exposing the private key or vulnerability to a MITM attack
> during the signing process.
> 
> 
> 
> Your help is appreciated.
> 
> 
> 
> --Pino

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] key_pair update on rebuild (a whole lot of conversations)

[Clint Byrum]

Excerpts from Sean Dague's message of 2017-10-03 16:16:48 -0400:
> There is currently a spec up for being able to specify a new key_pair
> name during the rebuild operation in Nova -
> https://review.openstack.org/#/c/375221/
> 
> For those not completely familiar with Nova operations, rebuild triggers
> the "reset this vm to initial state" by throwing out all the disks, and
> rebuilding them from the initial glance images. It does however keep the
> IP address and device models when you do that. So it's useful for
> ephemeral but repeating workloads, where you'd rather not have the
> network information change out from under you.
> 
> The spec is a little vague about when this becomes really useful,
> because this will not save you from "I lost my private key, and I have
> important data on that disk". Because the disk is destroyed. That's the
> point of rebuild. We once added this preserve_ephemeral flag to rebuild
> for trippleo on ironic, but it's so nasty we've scoped it to only work
> with ironic backends. Ephemeral should mean ephemeral.
> 

Let me take a moment to apologize for that feature. It was the worst idea
we had in TripleO, even worse than the name. ;)

> Rebuild bypasses the scheduler. A rebuilt server stays on the same host
> as it was before, which means the operation has a good chance of being
> faster than a DELETE + CREATE, as the image cache on that host should
> already have the base image for you instance.
> 

There are some pro's, but for the most part I'd rather train my users
to be creating new instances than train them to cling to fixed IPs and
single compute node resources. It's a big feature, and obviously we've
given it to users so they use it. But that doesn't mean it's the best
use of Nova development's time to be supporting it, nor is it the most
scalable way for users to interact with a cloud.

A trade-off for instance, is that a rebuilding server is unavailable while
rebuilding. The user cannot choose how long that server is unavailable,
or choose to roll back and make it available if something goes wrong. It's
rebuilding until it isn't. A new server, spun up somewhere else, can be
fully prepared before any switch is made. One of the best things about
being a cloud operator is that you put more onus on the users to fix
their own problems, and give them lots of tools to do it. But while a
server is being rebuilt it is entirely _the operator's problem_.

Also as an operator, while I appreciate that it's quick on that compute
node, I'd rather new servers be scheduled to the places that my scheduler
rules say they should go. I will at times want to drain a compute node,
and the longer the pet servers stick around and are rebuilt, the more
likely I am to have to migrate them forcibly.

> = Where I think we are? =
> 
> I think with all this data we're at the following:
> 
> Q: Should we add this to rebuild
> A: Yes, probably - after some enhancement to the spec *
> 
> * - we really should have much better use cases about the situations it
> is expected to be used in. We spend a lot of time 2 and 3 years out
> trying to figure out how anyone would ever use a feature, and adding
> another one without this doesn't seem good
> 
> Q: should this also be on reboot?
> A: NO - it would be too fragile
> 
> 
> I also think figuring out a way to get Nova out of the key storage
> business (which it really shouldn't be in) would be good. So if anyone
> wants to tackle Nova using Barbican for keys, that would be ++. Rebuild
> doesn't wait on that, but Barbican urls for keys seems like a much
> better world to be in.
> 

The keys are great. Barbican is a fantastic tool for storing _secret_
keys, but feels like a massive amount of overkill for this tiny blob of
public data.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptg] Simplification in OpenStack

[Clint Byrum]

Excerpts from Jonathan Proulx's message of 2017-09-26 16:01:26 -0400:
> On Tue, Sep 26, 2017 at 12:16:30PM -0700, Clint Byrum wrote:
> 
> :OpenStack is big. Big enough that a user will likely be fine with learning
> :a new set of tools to manage it.
> 
> New users in the startup sense of new, probably.
> 
> People with entrenched environments, I doubt it.
> 

Sorry no, I mean everyone who doesn't have an OpenStack already.

It's nice and all, if you're a Puppet shop, to get to use the puppet
modules. But it doesn't bring you any closer to the developers as a
group. Maybe a few use Puppet, but most don't. And that means you are
going to feel like OpenStack gets thrown over the wall at you once every
6 months.

> But OpenStack is big. Big enough I think all the major config systems
> are fairly well represented, so whether I'm right or wrong this
> doesn't seem like an issue to me :)
> 

They are. We've worked through it. But that doesn't mean potential
users are getting our best solution or feeling well integrated into
the community.

> Having common targets (constellations, reference architectures,
> whatever) so all the config systems build the same things (or a subset
> or superset of the same things) seems like it would have benefits all
> around.
> 

It will. It's a good first step. But I'd like to see a world where
developers are all well versed in how operators actually use OpenStack.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptg] Simplification in OpenStack

[Clint Byrum]

Excerpts from Samuel Cassiba's message of 2017-09-25 17:27:25 -0700:
> 
> > On Sep 25, 2017, at 16:52, Clint Byrum  wrote:
> > 
> > Excerpts from Jonathan D. Proulx's message of 2017-09-25 11:18:51 -0400:
> >> On Sat, Sep 23, 2017 at 12:05:38AM -0700, Adam Lawson wrote:
> >> 
> >> :Lastly, I do think GUI's make deployments easier and because of that, I
> >> :feel they're critical. There is more than one vendor whose built and
> >> :distributes a free GUI to ease OpenStack deployment and management. That's
> >> :a good start but those are the opinions of a specific vendor - not he OS
> >> :community. I have always been a big believer in a default cloud
> >> :configuration to ease the shock of having so many options for everything. 
> >> I
> >> :have a feeling however our commercial community will struggle with
> >> :accepting any method/project other than their own as being part a default
> >> :config. That will be a tough one to crack.
> >> 
> >> Different people have differnt needs, so this is not meant to
> >> contradict Adam.
> >> 
> >> But :)
> >> 
> >> Any unique deployment tool would be of no value to me as OpenStack (or
> >> anyother infrastructure component) needs to fit into my environment.
> >> I'm not going to adopt something new that requires a new parallel
> >> management tool to what I use.
> >> 
> > 
> > You already have that if you run OpenStack.
> > 
> > The majority of development testing and gate testing happens via
> > Devstack. A parallel management tool to what most people use to actually
> > operate OpenStack.
> > 
> >> I think focusing on the existing configuration management projects it
> >> the way to go. Getting Ansible/Puppet/Chef/etc.. to support a well
> >> know set of "constellations" in an opinionated would make deployment
> >> easy (for most people who are using one of those already) and ,
> >> ussuming the opionions are the same :) make consumption easier as
> >> well.
> >> 
> >> As an example when I started using OpenStack (Essex) we  had recently
> >> switch to Ubuntu as our Linux platform and Pupept as our config
> >> management. Ubuntu had a "one click MAAS install of OpenStack" which
> >> was impossible as it made all sorts of assumptions about our
> >> environment and wanted controll of most of them so it could provide a
> >> full deployemnt solution.  Puppet had a good integrated example config
> >> where I plugged in some local choices and and used existing deploy
> >> methodologies.
> >> 
> >> I fought with MAAS's "simple" install for a week.  When I gave up and
> >> went with Puppet I had live users on a substantial (for the time)
> >> cloud in less htan 2 days.
> >> 
> >> I don't think this has to do with the relative value of MASS and
> >> Puppet at the time, but rather what fit my existing deploy workflows.
> >> 
> >> Supporting multiple config tools may not be simple from an upstream
> >> perspective, but we do already have these projects and it is simpler
> >> to consume for brown field deployers at least.
> >> 
> > 
> > I don't think anybody is saying we would slam the door in the face of
> > people who use any one set of tools.
> > 
> > But rather, we'd start promoting and using a single solution for the bulk
> > of community efforts. Right now we do that with devstack as a reference
> > implementation that nobody should use for anything but dev/test. But
> > it would seem like a good idea for us to promote a tool for going live
> > as well.
> 
> Except by that very statement, you slam the door in the face of tons of 
> existing
> knowledge within organizations. This slope has a sheer face.
> 
> Promoting a single solution would do as much harm as it would good, for all 
> it’s
> worth. In such a scenario, the most advocated method would become the only
> understood method, in spite of all other deployment efforts. Each project that
> did not have the most mindshare would become more irrelevant than they are now
> and further slip into decay. For those that did not have the fortune or
> foresight to land on this hypothetical winning side, what for their efforts,
> evolve or gtfo?
> 
> I'm not saying Fuel or Salt or Chef or Puppet or Ansible needs to be the
> 'winner', because there isn't a competition, at least in my opinion. The way I
> see it, we

Re: [openstack-dev] [ptg] Simplification in OpenStack

[Clint Byrum]

Excerpts from Jonathan D. Proulx's message of 2017-09-25 11:18:51 -0400:
> On Sat, Sep 23, 2017 at 12:05:38AM -0700, Adam Lawson wrote:
> 
> :Lastly, I do think GUI's make deployments easier and because of that, I
> :feel they're critical. There is more than one vendor whose built and
> :distributes a free GUI to ease OpenStack deployment and management. That's
> :a good start but those are the opinions of a specific vendor - not he OS
> :community. I have always been a big believer in a default cloud
> :configuration to ease the shock of having so many options for everything. I
> :have a feeling however our commercial community will struggle with
> :accepting any method/project other than their own as being part a default
> :config. That will be a tough one to crack.
> 
> Different people have differnt needs, so this is not meant to
> contradict Adam.
> 
> But :)
> 
> Any unique deployment tool would be of no value to me as OpenStack (or
> anyother infrastructure component) needs to fit into my environment.
> I'm not going to adopt something new that requires a new parallel
> management tool to what I use.
> 

You already have that if you run OpenStack.

The majority of development testing and gate testing happens via
Devstack. A parallel management tool to what most people use to actually
operate OpenStack.

> I think focusing on the existing configuration management projects it
> the way to go. Getting Ansible/Puppet/Chef/etc.. to support a well
> know set of "constellations" in an opinionated would make deployment
> easy (for most people who are using one of those already) and ,
> ussuming the opionions are the same :) make consumption easier as
> well.
> 
> As an example when I started using OpenStack (Essex) we  had recently
> switch to Ubuntu as our Linux platform and Pupept as our config
> management. Ubuntu had a "one click MAAS install of OpenStack" which
> was impossible as it made all sorts of assumptions about our
> environment and wanted controll of most of them so it could provide a
> full deployemnt solution.  Puppet had a good integrated example config
> where I plugged in some local choices and and used existing deploy
> methodologies.
> 
> I fought with MAAS's "simple" install for a week.  When I gave up and
> went with Puppet I had live users on a substantial (for the time)
> cloud in less htan 2 days.
> 
> I don't think this has to do with the relative value of MASS and
> Puppet at the time, but rather what fit my existing deploy workflows.
> 
> Supporting multiple config tools may not be simple from an upstream
> perspective, but we do already have these projects and it is simpler
> to consume for brown field deployers at least.
> 

I don't think anybody is saying we would slam the door in the face of
people who use any one set of tools.

But rather, we'd start promoting and using a single solution for the bulk
of community efforts. Right now we do that with devstack as a reference
implementation that nobody should use for anything but dev/test. But
it would seem like a good idea for us to promote a tool for going live
as well.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptg] Simplification in OpenStack

[Clint Byrum]

Excerpts from Jeremy Stanley's message of 2017-09-21 16:17:00 +:
> On 2017-09-20 17:39:38 -0700 (-0700), Clint Byrum wrote:
> [...]
> > Something about common use cases and the exact mix of
> > projects + configuration to get there, and testing it? Help?
> [...]
> 
> Maybe you're thinking of the "constellations" suggestion? It found
> its way into the TC vision statement, though the earliest mention I
> can locate is in John's post to this ML thread:
> 
> http://lists.openstack.org/pipermail/openstack-dev/2017-April/115319.html
> 

Yes, constellations. Thanks!

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][nova][mogan] How to show respect to the original authors?

[Clint Byrum]

Excerpts from Michael Still's message of 2017-09-20 10:25:17 -0600:
> Dims, I'm not sure that's actually possible though. Many of these files
> have been through rewrites and developed over a large number of years.
> Listing all authors isn't practical.
> 
> Given the horse has bolted on forking these files, I feel like a comment
> acknowledging the original source file is probably sufficient.
> 
> What is concerning to me is that some of these files are part of the "ABI"
> of nova, and if mogan diverges from that then I think we're going to see
> user complaints in the future. Specifically configdrive, and metadata seem
> like examples of this. I don't want to see us end up in another "managed
> cut and paste" like early oslo where nova continues to develop these and
> mogan doesn't notice the changes.
> 
> I'm not sure how we resolve that. One option would be to refactor these
> files into a shared library.
> 

Agreed 100%. It would be better to have something completely different
than something that works 97% the same but constantly skews.

Luckily, since these things are part of the ABI of Nova, they are
versioned in many cases, and in all have a well defined interfaces on
one side. Seems like it should be relatively straight forward to wrap
the other side of them and call it a library.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ptg] Simplification in OpenStack

[Clint Byrum]

Wading in a bit late as I've been off-list for a while, but I have thoughts 
here.

Excerpts from Jay Pipes's message of 2017-09-13 13:44:55 -0400:
> On 09/12/2017 06:53 PM, Boris Pavlovic wrote:
> > Mike,
> > 
> > Great intiative, unfortunately I wasn't able to attend it, however I 
> > have some thoughts...
> > You can't simplify OpenStack just by fixing few issues that are 
> > described in the etherpad mostly..
> > 
> > TC should work on shrinking the OpenStack use cases and moving towards 
> > the product (box) complete solution instead of pieces of bunch barely 
> > related things..
> 
> OpenStack is not a product. It's a collection of projects that represent 
> a toolkit for various cloud-computing functionality.
>

I think Boris was suggesting that making it a product would simplify it.

I believe there is some effort under way to try this, but my brain
has ceased to remember what that effort is called or how it is being
implemented. Something about common use cases and the exact mix of
projects + configuration to get there, and testing it? Help?

> > *Simple things to improve: *
> > /This is going to allow community to work together, and actually get 
> > feedback in standard way, and incrementally improve quality. /
> > 
> > 1) There should be one and only one:
> > 1.1) deployment/packaging(may be docker) upgrade mechanism used by 
> > everybody
> 
> Good luck with that :) The likelihood of the deployer/packager community 
> agreeing on a single solution is zero.
> 

I think Boris is suggesting that the OpenStack development community
pick one to use, not the packaging and deployer community. The
only common thing dev has in this area is devstack, and that
has allowed dev to largely ignore issues they create because
they're not feeling the pain of the average user who is using
puppet/chef/ansible/tripleo/kolla/in-house-magic to deploy.

> > 1.2) monitoring/logging/tracing mechanism used by everybody
> 
> Also close to zero chance of agreeing on a single solution. Better to 
> focus instead on ensuring various service projects are monitorable and 
> transparent.
> 

I'm less enthused about this one as well. Monitoring, alerting, defining
business rules for what is broken and what isn't are very org-specific
things.

I also don't think OpenStack fails at this and there is plenty exposed
in clear ways for monitors to be created.

> > 1.3) way to configure all services (e.g. k8 etcd way)
> 
> Are you referring to the way to configure k8s services or the way to 
> configure/setup an *application* that is running on k8s? If the former, 
> then there is *not* a single way of configuring k8s services. If the 
> latter, there isn't a single way of configuring that either. In fact, 
> despite Helm being a popular new entrant to the k8s application package 
> format discussion, k8s itself is decidedly *not* opinionated about how 
> an application is configured. Use a CMDB, use Helm, use env variables, 
> use confd, use whatever. k8s doesn't care.
> 

We do have one way to configure things. Well.. two.

*) Startup-time things are configured in config files.
*) Run-time changable things are in databases fronted by admin APIs/tools.

> > 2) Projects must have standardize interface that allows these projects 
> > to use them in same way.
> 
> Give examples of services that communicate over *non-standard* 
> interfaces. I don't know of any.
> 

Agreed here too. I'd like to see a more clear separation between nova,
neutron, and cinder on the hypervisor, but the way they're coupled now
*is* standardized.

> > 3) Testing & R&D should be performed only against this standard deployment
> 
> Sorry, this is laughable. There will never be a standard deployment 
> because there are infinite use cases that infrastructure supports. 
> *Your* definition of what works for GoDaddy is decidedly different from 
> someone else's definition of what works for them.
> 

If there were a few well defined product definitions, there could be. It's
not laughable at all to me. devstack and the configs it creates are useful
for lightweight testing, but they're not necessarily representative of
the standard makeup of real-world clouds.

> > *Hard things to improve: *
> > 
> > OpenStack projects were split in far from ideal way, which leads to 
> > bunch of gaps that we have now:
> > 1.1) Code & functional duplications:  Quotas, Schedulers, Reservations, 
> > Health checks, Loggign, Tracing, 
> 
> There is certainly code duplication in some areas, yes.
> 

I feel like this de-duplication has been moving at the slow-but-consistent
pace anyone can hope for since it was noticed and oslo was created.

It's now at the things that are really hard to de-dupe like quotas and policy.

> > 1.2) Non optimal workflows (booting VM takes 400 DB requests) because 
> > data is stored in Cinder,Nova,Neutron
> 
> Sorry, I call bullshit on this. It does not take 400 DB requests to boot 
> a VM. Also: the DB is not at all the bottleneck in the VM launc

Re: [openstack-dev] [glance] docs: architecture or hw_architecture

[Clint Byrum]

Excerpts from Dean Troyer's message of 2017-08-23 13:48:07 -0500:
> On Wed, Aug 23, 2017 at 12:33 PM, Brian Rosmaita
>  wrote:
> > My point is just that Glance went one way on this, Nova went a
> > different way, and users are left hanging.  My secondary point is that
> > it might be useful to catalog these things in the metadata definitions
> > service so that we all stay on the same page better.
> 
> This is a great example of why attributes that are actually used
> elsewhere should be defined in the API rather than just picked out of
> a collection of metadata/properties.

Note that there was at least an effort at one time to create a
definitive place to share tags across OpenStack projects and across
clouds:

https://wiki.openstack.org/wiki/Graffiti

I don't know where this is at these days, but I would agree that I'd
rather see any guarantees a service provides documented and expressed in
its API. When one project's objects are referenced in anothers' API,
that's a good place for those two projects to work very closely
together. It looks like that didn't quite happen here. A quick doc fix
should resolve the matter, or is the use of 'architecture' now in other
places?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra]Plan to support Python 3.6 ?

[Clint Byrum]

Ubuntu will ship Python 3.6 in the next LTS, Ubuntu 18.04 next April.

AFAICT CentOS 7 still doesn't ship Python 3 in mainline. EPEL has 3.4
still.

Excerpts from Doug Hellmann's message of 2017-08-09 09:31:37 -0400:
> Excerpts from ChangBo Guo's message of 2017-08-09 21:25:07 +0800:
> > We received Python 3.6 related Bug recently [1][2]. That let me think
> > what's the plan to support Python 3.6 for OpenStack in the future.   Python
> > 3.6 was released on December 23, 2016, has some different behaviors from
> > Python 3.5[3]. talked with cdent in the IRC, would like to discuss this
> > through mailing list , and suggest a discussion at the PTG[3]
> 
> One of the reasons we were able to move ahead with 3.5 was that it would
> be available on the platforms for which we test deployments, as defined
> in the CTI [5]. Are Ubuntu LTS and CentOS shipping Python 3.6?
> 
> [5] 
> https://governance.openstack.org/tc/reference/project-testing-interface.html
> 
> > 
> > 1. what's the time to support Python 3.6 ?
> > 
> > 2. what 's the  plan or process ?
> >  Maybe we can do that we support python 3
> >  1) setup python 3.6 non-voting CI jobs
> >  2) Fix related issues about Python 3.6
> >  3) enable pyhton 3.6 jobs voting
> > 
> > [1] https://bugs.launchpad.net/oslo.rootwrap/+bug/1709505
> > [2]https://bugs.launchpad.net/pbr/+bug/1690103
> > [3]https://docs.python.org/3/whatsnew/3.6.html
> > [4] https://etherpad.openstack.org/p/infra-ptg-queens
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [python-openstackclient][python-openstacksdk][neutron][nova] supporting resource extensions with our CLI

[Clint Byrum]

Excerpts from Matt Riedemann's message of 2017-08-07 14:33:22 -0500:
> On 8/3/2017 1:39 PM, Boden Russell wrote:
> > I think we have a gap in our OSC CLI for non-stadium plugin/driver
> > projects (neutron plugin projects, nova driver projects) that implement
> > RESTful resource API attribute extensions.
> > 
> > For details, go directly to [1].
> > For a summary read on...
> > 
> > 
> > For OpenStack APIs that support extensions (ex [2]), the classic
> > python-client CLIs worked "out of the box" for extensions
> > adding attributes to existing RESTful resources.
> > 
> > For example, your project has a neutron plugin that adds a 'my_bool'
> > boolean attribute to 'network' resources that can be set via POST/PUT
> > and is returned with GET. This just works with the python-neutronclient
> > CLI without any client-side code changes.
> > 
> > 
> > However, with OSC resource attributes must be added directly/statically
> > to the sdk's resource and then consumed in the client; the support does
> > not come "for free" in the CLI. While this is fine for stadium projects
> > (they can contribute directly to the sdk/client), non-stadium projects
> > have no viable option to plugin/extend the CLI today for this type of
> > API extension mechanism.
> > 
> > With the phasing out of the python clients, a number of our users will
> > be left without a CLI to interface with these extensions.
> > 
> > I'd like to try and close this gap in Queens and welcome discussion in [1].
> > 
> > Thanks
> > 
> > 
> > [1] https://bugs.launchpad.net/python-openstacksdk/+bug/1705755
> > [2] https://wiki.openstack.org/wiki/NeutronDevelopment#API_Extensions
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> There is nothing like this for Nova so I'm not sure why Nova should be 
> involved here. We dropped all support for extending the API via 
> stevedore extension loading in Pike [1]. The virt drivers don't extend 
> the REST API either.
> 
> [1] https://blueprints.launchpad.net/nova/+spec/api-no-more-extensions-pike
> 

And there was much rejoicing.

If the thing you're doing doesn't fit in the mainline API, then what
you're doing is making a new API. Extensions just bypass the important
part where that API gets designed and thought through.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [TripleO] Let's use Ansible to deploy OpenStack services on Kubernetes

[Clint Byrum]

Excerpts from Bogdan Dobrelya's message of 2017-07-14 18:14:42 +0200:
> On 14.07.2017 17:55, Michał Jastrzębski wrote:
> > Guys you just described Kolla-Kubernetes pretty much... how about
> > we join effort and work towards this goal together?
> 
> That's exactly that I'd like we all to do.
> 

Agreed, and ...

> > 
> > On 14 July 2017 at 08:43, Flavio Percoco  wrote:
> >> On 14/07/17 17:26 +0200, Bogdan Dobrelya wrote:
> >>>
> >>> On 14.07.2017 11:17, Flavio Percoco wrote:
> 
> 
>  Greetings,
> 
>  As some of you know, I've been working on the second phase of TripleO's
>  containerization effort. This phase if about migrating the docker based
>  deployment onto Kubernetes.
> 
>  These phase requires work on several areas: Kubernetes deployment,
>  OpenStack
>  deployment on Kubernetes, configuration management, etc. While I've been
>  diving
>  into all of these areas, this email is about the second point, OpenStack
>  deployment on Kubernetes.
> 
>  There are several tools we could use for this task. kolla-kubernetes,
>  openstack-helm, ansible roles, among others. I've looked into these
>  tools and
>  I've come to the conclusion that TripleO would be better of by having
>  ansible
>  roles that would allow for deploying OpenStack services on Kubernetes.
> 
>  The existing solutions in the OpenStack community require using Helm.
>  While I
>  like Helm and both, kolla-kubernetes and openstack-helm OpenStack
>  projects, I
>  believe using any of them would add an extra layer of complexity to
>  TripleO,
> >>>
> >>>
> >>> It's hard to estimate that complexity w/o having a PoC of such an
> >>> integration. We should come up with a final choice once we have it done.
> >>>
> >>> My vote would go for investing engineering resources into solutions that
> >>> have problems already solved, even by the price of added complexity (but
> >>> that sort of depends...). Added complexity may be compensated with
> >>> removed complexity (like those client -> Mistral -> Heat -> Mistral ->
> >>> Ansible manipulations discussed in the mail thread mentioned below [0])
> >>
> >>
> >> I agree it's hard to estimate but you gotta draw the line somewhere. I
> >> actually
> >> spent time on this and here's a small PoC of ansible+mariadb+helm. I wrote
> >> the
> >> pyhelm lib (took some code from the openstack-helm folks) and I wrote the
> >> ansible helm module myself. I'd say I've spent enough time on this 
> >> research.
> >>
> >> I don't think getting a full PoC working is worth it as that will require
> >> way
> >> more work for not much value since we can anticipate some of the
> >> complexities
> >> already.
> >>
> >> As far as the complexity comment goes, I disagree with you. I don't think
> >> you're
> >> evaluating the amount of complexity that there *IS* already in TripleO and
> >> how
> >> adding more complexity (layers, states, services) would make things worse
> >> for
> >> not much extra value.
> >>
> >> By all means, I might be wrong here so, do let me know if you're seeing
> >> something I'm not.
> 
> My point was to "trade" complexity described in the "Forming our plans
> around Ansible​" ML thread:
> 
> (3) Mistral calling Heat calling Mistral calling Ansible
> 
> to just
> 
> (3') something calls kolla-kubernetes/openstack-helm, via some wrapper
> composition overlay (which creates complexity), or the like
> 
> While the latter might add complexity like the way you (Flavio) have
> described, the former would remove *another* type of complexity, and the
> result might worth the efforts.
>

The two options seem to be

a. Bootstrap helm and charts and then use openstack-helm/kolla-kubernetes
b. Bootstrap (something) and then use newly minted ansible to manipulate
   kubernetes.

(a) seems like less net complexity, as bootstrapping code is usually
able to be more naive. The new ansible will have to be at least as good
as openstack-helm and kolla-kubernetes, and still needs bootstraps of
its own.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [congress] Using congress to improve the consistency of configuration files.

[Clint Byrum]

Excerpts from valentin.matton's message of 2017-07-04 15:29:25 +0200:
> We would like to use congress to check the consistency of the 
> configuration files used by the various Openstack services on different 
> nodes.
> 
> Although installers do a great job for ensuring that the initial 
> definition of those files are correct, it may be necessary to tweak 
> those files on running instances
> or to use templates that are out of the bounds of the pre-configured 
> use-cases. Then the administrator must modify the configuration without 
> any safety net.
> 

While I'm sure this does still happen, you'd have to be living under a
rock to think that this is the state of the art.

I'm certain _most_ OpenStack installs are using automated configuration
management to assert their config file content. And if they're
practicing safe deploys, they also have integration tests to make sure
those modifications work properly.

Now, if Congress does in fact want to compete with Puppet / Chef /
Ansible / Salt / etc., I suppose that's Congress's choice. But just to
be clear: very few operators are doing what you suggest as the reason
to make Congress do this.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][tc] How to deal with confusion around "hosted projects"

[Clint Byrum]

This message makes a bunch of salient, valid points, none of which I
wish to directly address.

However, on the whole, I think the analysis stops short of pushing through
to a root cause, and thus, the solution proposed is entirely focused on
symptoms.

The root cause of all of this until now has been not really knowing what
OpenStack is. The visioning recently done was a great step in the right
direction toward this. I would like to make sure that we acknowledge
this while we address symptoms of the past choices we made.

In particular, I wonder if landing the vision, pushing harder to fill
in any missing parts of the constellation concept, and getting actual
constellations defined _immediately_, would do just as much as drawing
these concerte lines around abstract bits of software.

So, I'm not saying any of this is wrong. I like the solution proposed,
and I think all of the problems stated are in fact problems. I just
wonder if we're bouncing off "visions are hard" and falling into a bit
of yak shaving around the easy problems when we as a community should
remain focused on the vision.

If nothing else, I'd like to see it clearly stated how this solution
fits in with the vision.

Excerpts from Thierry Carrez's message of 2017-06-28 16:50:01 +0200:
> Hi everyone,
> 
> Two weeks ago, as a result of a discussion at the Board+TC+UC workgroup
> working on "better communicating what is openstack", I started a
> thread[1] about moving away from big tent terminology. The thread went
> in a lot of directions, including discussing GitHub mirroring strategy,
> what makes projects want to be official, the need for returning to a
> past when everything was (supposedly) simpler, and naming fun.
> 
> [1] http://lists.openstack.org/pipermail/openstack-dev/2017-June/118368.html
> 
> Many agreed that the "Big Tent" name (as a synonym to "official
> openstack projects") is hurting more than it helps, and we should stop
> using it. The issue is, merely stopping using it won't be enough. We
> have tried, and the name sticks. You need to replace the name by
> something that sticks more, or address the root cause directly.
> 
> The central issue being discussed here is an issue of external
> perception. It's hard for newcomers to the OpenStack world to see what
> is a part of OpenStack and what's not. If you google "openstack machine
> learning", the first hits are Cognitive and Meteos, and it's impossible
> to tell that those are actually not OpenStack projects. One of those has
> been dead for 2 years -- having people think that those are official
> projects hurts all the OpenStack projects, by lowering expectations
> around what that means, in terms of quality, maintenance, or community.
> 
> The confusion mainly stems from the fact that OpenStack project
> infrastructure is open to any open source project (and it's nobody's job
> to clean up dead things). So you can find (on our wiki, on our
> mailing-list, on our cgit farm, on our gerrit, on our GitHub
> organization...) things that are actually not OpenStack, even with the
> expansive "are you one of us" definition. Arguably the most confusing
> aspect is the "openstack/" prefix in the git repository name, which
> indicates some kind of brand association.
> 
> I'd say we have two options. We can address the perception issue on the
> edges, or you can treat the root cause. Neither of those options is
> really an OpenStack  governance change (or "big tent" change) -- they
> are more about what to do with things that are actually *not* in our
> governance.
> 
> Addressing the perception on the edges means making it clearer when
> things are not official. The thread above discussed a lot of potential
> solutions. We could give unofficial things a catchy group name
> (Stackforge, Opium, Electrons...), and hope it sticks. We could find a
> way to tag all projects on GitHub that are not official, or mirror them
> to another organization, or stop mirroring them altogether. We could
> remove the openstack/ prefix from all the projects we host. We could
> actively mark all wiki pages that are not about an official project. We
> could set up a separate Gerrit or separate ML for hosted projects
> development discussions. The main issue with that approach is that it's
> a *lot* of work, and it never completely eliminates the confusion.
> 
> Removing the root cause would be a more radical move: stop offering
> hosting to non-OpenStack projects on OpenStack infrastructure
> altogether. We originally did that for a reason, though. The benefits of
> offering that service are:
> 
> 1- it lets us set up code repositories and testing infrastructure before
> a project applies to be an official OpenStack project.
> 
> 2- it lets us host things that are not openstack but which we work on
> (like abandoned Python libraries or GPL-licensed things) in a familiar
> environment
> 
> 3- it spreads "the openstack way" (Gerrit, Zuul) beyond openstack itself
> 
> I would argue that we could handle (1)

Re: [openstack-dev] [trove][all][tc] A proposal to rearchitect Trove

[Clint Byrum]

tl;dr - I think Trove's successor has a future, but there are two
conflicting ideas presented and Trove should pick one or the other.

Excerpts from Amrith Kumar's message of 2017-06-18 07:35:49 -0400:
> 
> We have learned a lot from v1, and the hope is that we can address that in
> v2. Some of the more significant things that I have learned are:
> 
> - We should adopt a versioned front-end API from the very beginning; making
> the REST API versioned is not a ‘v2 feature’
> 

+1

> - A guest agent running on a tenant instance, with connectivity to a shared
> management message bus is a security loophole; encrypting traffic,
> per-tenant-passwords, and any other scheme is merely lipstick on a security
> hole
>

This is a broad statement, and I'm not sure I understand the actual risk
you're presenting here as "a security loophole".

How else would you administer a database server than through some kind
of agent? Whether that agent is a python daemon of our making, sshd, or
whatever kubernetes component lets you change things, they're all
administrative pieces that sit next to the resource.

> - Reliance on Nova for compute resources is fine, but dependence on Nova VM
> specific capabilities (like instance rebuild) is not; it makes things like
> containers or bare-metal second class citizens
> 

I whole heartedly agree that rebuild is a poor choice for database
servers. In fact, I believe it is a completely non-scalable feature that
should not even exist in Nova.

This is kind of a "we shouldn't be this". What should we be running
database clusters on?

> - A fair portion of what Trove does is resource orchestration; don’t
> reinvent the wheel, there’s Heat for that. Admittedly, Heat wasn’t as far
> along when Trove got started but that’s not the case today and we have an
> opportunity to fix that now
> 

Yeah. You can do that. I'm not really sure what it gets you at this
level. There was an effort a few years ago to use Heat for Trove and
some other pieces, but they fell short at the point where they had to
ask Heat for a few features like, oddly enough, rebuild confirmation
after test. Also, it increases friction to your project if your project
requires Heat in a cloud. That's a whole new service that one would have
to choose to expose or not to users and manage just for Trove. That's
a massive dependency, and it should come with something significant. I
don't see what it actually gets you when you already have to keep track
of your resources for cluster membership purposes anyway.

> - A similarly significant portion of what Trove does is to implement a
> state-machine that will perform specific workflows involved in implementing
> database specific operations. This makes the Trove taskmanager a stateful
> entity. Some of the operations could take a fair amount of time. This is a
> serious architectural flaw
> 

A state driven workflow is unavoidable if you're going to do cluster
manipulation. So you can defer this off to Mistral or some other
workflow engine, but I don't think it's an architectural flaw _that
Trove does it_. Clusters have states. They have to be tracked. Do that
well and your users will be happy.

> - Tenants should not ever be able to directly interact with the underlying
> storage and compute used by database instances; that should be the default
> configuration, not an untested deployment alternative
> 

Agreed. There's no point in having an "inside the cloud" service if
you're just going to hand them the keys to the VMs and volumes anyway.

The point of something like Trove is to be able to retain control at the
operator level, and only give users the interface you promised,
optimized without the limitations of the cloud.

> - The CI should test all databases that are considered to be ‘supported’
> without excessive use of resources in the gate; better code modularization
> will help determine the tests which can safely be skipped in testing changes
> 

Take the same approach as the other driver-hosting things. If it's
in-tree, it has to have a gate test.

> - Clusters should be first class citizens not an afterthought, single
> instance databases may be the ‘special case’, not the other way around
> 

+1

> - The project must provide guest images (or at least complete tooling for
> deployers to build these); while the project can’t distribute operating
> systems and database software, the current deployment model merely impedes
> adoption
> 

IIRC the project provides dib elements and a basic command line to build
images for your cloud, yes? Has that not worked out?

> - Clusters spanning OpenStack deployments are a real thing that must be
> supported
> 

This is the most problematic thing you asserted. There are two basic
desires I see that drive a Trove adoption:

1) I need database clusters and I don't know how to do them right.
2) I need _high performance/availability/capacity_ databases and my
cloud's standard VM flavors/hosts/networks/disks/etc. stand in the way
of that.

For t

Re: [openstack-dev] [trove][all][tc] A proposal to rearchitect Trove

[Clint Byrum]

Excerpts from Jay Pipes's message of 2017-06-20 10:08:54 -0400:
> On 06/20/2017 09:42 AM, Doug Hellmann wrote:
> > Does "service VM" need to be a first-class thing?  Akanda creates
> > them, using a service user. The VMs are tied to a "router" which
> > is the billable resource that the user understands and interacts with
> > through the API.
> 
> Frankly, I believe all of these types of services should be built as 
> applications that run on OpenStack (or other) infrastructure. In other 
> words, they should not be part of the infrastructure itself.
> 
> There's really no need for a user of a DBaaS to have access to the host 
> or hosts the DB is running on. If the user really wanted that, they 
> would just spin up a VM/baremetal server and install the thing themselves.
> 

There's one reason, and that is specialized resources that we don't
trust to be multi-tenant.

Baremetal done multi-tenant is hard, just ask our friends who were/are
running OnMetal. But baremetal done for the purposes of running MySQL
clusters that only allow users to access MySQL and control everything
via an agent of sorts is a lot simpler. You can let them all share a
layer 2 with no MAC filtering for instance, since you are in control at
the OS level.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [swift] Optimizing storage for small objects in Swift

[Clint Byrum]

Excerpts from Alexandre Lécuyer's message of 2017-06-19 11:36:15 +0200:
> Hello Clint,
> 
> Thanks for your feedback, replying in the email inline.
> 
> On 06/16/2017 10:54 PM, Clint Byrum wrote:
> > Excerpts from John Dickinson's message of 2017-06-16 11:35:39 -0700:
> >> On 16 Jun 2017, at 10:51, Clint Byrum wrote:
> >>
> >>> This is great work.
> >>>
> >>> I'm sure you've already thought of this, but could you explain why
> >>> you've chosen not to put the small objects in the k/v store as part of
> >>> the value rather than in secondary large files?
> >> I don't want to co-opt an answer from Alex, but I do want to point to some 
> >> of the other background on this LOSF work.
> >>
> >> https://wiki.openstack.org/wiki/Swift/ideas/small_files
> >> https://wiki.openstack.org/wiki/Swift/ideas/small_files/experimentations
> >> https://wiki.openstack.org/wiki/Swift/ideas/small_files/implementation
> >>
> > These are great. Thanks for sharing them, I understand a lot more now.
> >
> >> Look at the second link for some context to your answer, but the summary 
> >> is "that means writing a file system, and writing a file system is really 
> >> hard".
> >>
> > I'm not sure we were thinking the same thing.
> >
> > I was more asking, why not put the content of the object into the k/v
> > instead of the big_file_id:offset? My thinking was that for smaller
> > objects, you would just return the data immediately upon reading the k/v,
> > rather than then needing to go find the big file and read the offset.
> > However, I'm painfully aware that those directly involved with the problem
> > have likely thought of this. However, the experiments don't seem to show
> > that this was attempted. Perhaps I'm zooming too far out to see the real
> > problem space. You can all tell me to take my spray paint can and stop
> > staring at the bike shed if this is just too annoying. Seriously.
> >
> > Of course, one important thing is, what does one consider "small"? Seems
> > like there's a size where the memory footprint of storing it in the
> > k/v would be justifiable if reads just returned immediately from k/v
> > vs. needing to also go get data from a big file on disk. Perhaps that
> > size is too low to really matter. I was hoping that this had been
> > considered and there was documentation, but I don't really see it.
> Right, we had considered this when we started the project : storing 
> small objects directly in the KV. It would not be too diffcult to do, 
> but we see a few problems :
> 
> 1) consistency
> In the current design, we append data at the end of a "big file". When 
> the data upload is finished, swift writes the metadata and commits the 
> file. This triggers a fsync(). Only then do we return. We can rely on 
> the data being stable on disk, even if there is a power loss.  Because 
> we fallocate() space for the "big files" beforehand, we can also hope to 
> have mostly sequential disk IO.
> (Important as most swift clusters use SATA disks).
> 
> Once the object has been committed, we create an entry for it in the KV. 
> This is done asynchronously, because synchronous writes on the KV kills 
> performance. If we loose power, we loose the latest data. After the 
> server is rebooted, we have to scan the end of volumes to create missing 
> entries in the KV. (I will not discuss this in detail in this email to 
> keep this short, but we can discuss it in another thread, or I can post 
> some information on the wiki).
> 
> If we put small objects in the KV, we would need to do synchronous 
> writes to make sure we don't loose data.
> Also, currently we can completly reconstruct the KV from the "big 
> files". It would not be possible anymore.
> 
> 
> 2) performance
> On our clusters we see about 40% of physical disk IO being caused by 
> readdir().
> We want to serve directory listing requests from memory. So "small" 
> means "the KV can fit in the page cache".
> We estimate that we need the size per object to be below 50 bytes, which 
> doesn't leave much room for data.
> 
> LevelDB causes write amplification, as it will regularly copy data to 
> different files (levels) to keep keys compressed and in sorted order. If 
> we store object data within the KV, it will be copied around multiple 
> times as well.
> 
> 
> Finally it is also more simple to have only one path to handle. Beyond 
> these issues, it would not be 

Re: [openstack-dev] [swift] Optimizing storage for small objects in Swift

[Clint Byrum]

Excerpts from John Dickinson's message of 2017-06-16 11:35:39 -0700:
> 
> On 16 Jun 2017, at 10:51, Clint Byrum wrote:
> 
> > This is great work.
> >
> > I'm sure you've already thought of this, but could you explain why
> > you've chosen not to put the small objects in the k/v store as part of
> > the value rather than in secondary large files?
> 
> I don't want to co-opt an answer from Alex, but I do want to point to some of 
> the other background on this LOSF work.
> 
> https://wiki.openstack.org/wiki/Swift/ideas/small_files
> https://wiki.openstack.org/wiki/Swift/ideas/small_files/experimentations
> https://wiki.openstack.org/wiki/Swift/ideas/small_files/implementation
> 

These are great. Thanks for sharing them, I understand a lot more now.

> Look at the second link for some context to your answer, but the summary is 
> "that means writing a file system, and writing a file system is really hard".
> 

I'm not sure we were thinking the same thing.

I was more asking, why not put the content of the object into the k/v
instead of the big_file_id:offset? My thinking was that for smaller
objects, you would just return the data immediately upon reading the k/v,
rather than then needing to go find the big file and read the offset.
However, I'm painfully aware that those directly involved with the problem
have likely thought of this. However, the experiments don't seem to show
that this was attempted. Perhaps I'm zooming too far out to see the real
problem space. You can all tell me to take my spray paint can and stop
staring at the bike shed if this is just too annoying. Seriously.

Of course, one important thing is, what does one consider "small"? Seems
like there's a size where the memory footprint of storing it in the
k/v would be justifiable if reads just returned immediately from k/v
vs. needing to also go get data from a big file on disk. Perhaps that
size is too low to really matter. I was hoping that this had been
considered and there was documentation, but I don't really see it.

Also the "writing your own filesystem" option in experiments seemed
more like a thing to do if you left the k/v stores out entirely.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [swift] Optimizing storage for small objects in Swift

[Clint Byrum]

This is great work.

I'm sure you've already thought of this, but could you explain why
you've chosen not to put the small objects in the k/v store as part of
the value rather than in secondary large files?

Excerpts from Alexandre Lécuyer's message of 2017-06-16 15:54:08 +0200:
> Swift stores objects on a regular filesystem (XFS is recommended), one file 
> per object. While it works fine for medium or big objects, when you have lots 
> of small objects you can run into issues: because of the high count of inodes 
> on the object servers, they can’t stay in cache, implying lot of memory usage 
> and IO operations to fetch inodes from disk.
> 
> In the past few months, we’ve been working on implementing a new storage 
> backend in Swift. It is highly inspired by haystack[1]. In a few words, 
> objects are stored in big files, and a Key/Value store provides information 
> to locate an object (object hash -> big_file_id:offset). As the mapping in 
> the K/V consumes less memory than an inode, it is possible to keep all 
> entries in memory, saving many IO to locate the object. It also allows some 
> performance improvements by limiting the XFS meta updates (e.g.: almost no 
> inode updates as we write objects by using fdatasync() instead of fsync())
> 
> One of the questions that was raised during discussions about this design is: 
> do we want one K/V store per device, or one K/V store per Swift partition (= 
> multiple K/V per device). The concern was about failure domain. If the only 
> K/V gets corrupted, the whole device must be reconstructed. Memory usage is a 
> major point in making a decision, so we did some benchmark.
> 
> The key-value store is implemented over LevelDB.
> Given a single disk with 20 million files (could be either one object replica 
> or one fragment, if using EC)
> 
> I have tested three cases :
>- single KV for the whole disk
>- one KV per partition, with 100 partitions per disk
>- one KV per partition, with 1000 partitions per disk
> 
> Single KV for the disk :
>- DB size: 750 MB
>- bytes per object: 38
> 
> One KV per partition :
> Assuming :
>- 100 partitions on the disk (=> 100 KV)
>- 16 bits part power (=> all keys in a given KV will have the same 16 bit 
> prefix)
> 
>- 7916 KB per KV, total DB size: 773 MB
>- bytes per object: 41
> 
> One KV per partition :
> Assuming :
>- 1000 partitions on the disk (=> 1000 KV)
>- 16 bits part power (=> all keys in a given KV will have the same 16 bit 
> prefix)
> 
>- 1388 KB per KV, total DB size: 1355 MB total
>- bytes per object: 71
>
> 
> A typical server we use for swift clusters has 36 drives, which gives us :
> - Single KV : 26 GB
> - Split KV, 100 partitions : 28 GB (+7%)
> - Split KV, 1000 partitions : 48 GB (+85%)
> 
> So, splitting seems reasonable if you don't have too many partitions.
> 
> Same test, with 10 million files instead of 20
> 
> - Single KV : 13 GB
> - Split KV, 100 partitions : 18 GB (+38%)
> - Split KV, 1000 partitions : 24 GB (+85%)
> 
> 
> Finally, if we run a full compaction on the DB after the test, you get the
> same memory usage in all cases, about 32 bytes per object.
> 
> We have not made enough tests to know what would happen in production. LevelDB
> does trigger compaction automatically on parts of the DB, but continuous 
> change
> means we probably would not reach the smallest possible size.
> 
> 
> Beyond the size issue, there are other things to consider :
> File descriptors limits : LevelDB seems to keep at least 4 file descriptors 
> open during operation.
> 
> Having one KV per partition also means you have to move entries between KVs 
> when you change the part power. (if we want to support that)
> 
> A compromise may be to split KVs on a small prefix of the object's hash, 
> independent of swift's configuration.
> 
> As you can see we're still thinking about this. Any ideas are welcome !
> We will keep you updated about more "real world" testing. Among the tests we 
> plan to check how resilient the DB is in case of a power loss.
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] etcd3 as base service - update

[Clint Byrum]

Excerpts from Lance Bragstad's message of 2017-06-08 16:10:00 -0500:
> On Thu, Jun 8, 2017 at 3:21 PM, Emilien Macchi  wrote:
> 
> > On Thu, Jun 8, 2017 at 7:34 PM, Lance Bragstad 
> > wrote:
> > > After digging into etcd a bit, one place this might be help deployer
> > > experience would be the handling of fernet keys for token encryption in
> > > keystone. Currently, all keys used to encrypt and decrypt tokens are
> > kept on
> > > disk for each keystone node in the deployment. While simple, it requires
> > > operators to perform rotation on a single node and then push, or sync,
> > the
> > > new key set to the rest of the nodes. This must be done in lock step in
> > > order to prevent early token invalidation and inconsistent token
> > responses.
> >
> > This is what we discussed a few months ago :-)
> >
> > http://lists.openstack.org/pipermail/openstack-dev/2017-March/113943.html
> >
> > I'm glad it's coming back ;-)
> >
> 
> Yep! I've proposed a pretty basic spec to backlog [0] in an effort to
> capture the discussion. I've also noted the point Kevin brought up about
> authorization in etcd (thanks, Kevin!)
> 
> If someone feels compelled to take that and run with it, they are more than
> welcome to.
> 
> [0] https://review.openstack.org/#/c/472385/
> 

I commented on the spec. I think this is a misguided idea. etcd3 is a
_coordination_ service. Not a key manager. It lacks the audit logging
and access control one expects to protect and manage key material. I'd
much rather see something like Hashicorp's Vault [1] implemented for
Fernet keys than etcd3. We even have a library for such things called
Castellan[2].

[1] https://www.vaultproject.io/
[2] https://docs.openstack.org/developer/castellan/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Cockroachdb for Keystone Multi-master

[Clint Byrum]

Excerpts from Jay Pipes's message of 2017-05-30 21:06:59 -0400:
> On 05/30/2017 05:07 PM, Clint Byrum wrote:
> > Excerpts from Jay Pipes's message of 2017-05-30 14:52:01 -0400:
> >> Sorry for the delay in getting back on this... comments inline.
> >>
> >> On 05/18/2017 06:13 PM, Adrian Turjak wrote:
> >>> Hello fellow OpenStackers,
> >>>
> >>> For the last while I've been looking at options for multi-region
> >>> multi-master Keystone, as well as multi-master for other services I've
> >>> been developing and one thing that always came up was there aren't many
> >>> truly good options for a true multi-master backend.
> >>
> >> Not sure whether you've looked into Galera? We had a geo-distributed
> >> 12-site Galera cluster servicing our Keystone assignment/identity
> >> information WAN-replicated. Worked a charm for us at AT&T. Much easier
> >> to administer than master-slave replication topologies and the
> >> performance (yes, even over WAN links) of the ws-rep replication was
> >> excellent. And yes, I'm aware Galera doesn't have complete snapshot
> >> isolation support, but for Keystone's workloads (heavy, heavy read, very
> >> little write) it is indeed ideal.
> >>
> > 
> > This has not been my experience.
> > 
> > We had a 3 site, 9 node global cluster and it was _extremely_ sensitive
> > to latency. We'd lose even read ability whenever we had a latency storm
> > due to quorum problems.
> > 
> > Our sites were London, Dallas, and Sydney, so it was pretty common for
> > there to be latency between any of them.
> > 
> > I lost track of it after some reorgs, but I believe the solution was
> > to just have a single site 3-node galera for writes, and then use async
> > replication for reads. We even helped land patches in Keystone to allow
> > split read/write host configuration.
> 
> Interesting, thanks for the info. Can I ask, were you using the Galera 
> cluster for read-heavy data like Keystone identity/assignment storage? 
> Or did you have write-heavy data mixed in (like Keystone's old UUID 
> token storage...)
> 
> It should be noted that CockroachDB's documentation specifically calls 
> out that it is extremely sensitive to latency due to the way it measures 
> clock skew... so might not be suitable for WAN-separated clusters?
> 

That particular Galera was for Keystone only, and we were using Fernet
tokens.

Revocation events were a constant but manageable source of writes. I
believe some optimizations were made to reduce the frequency of the events
but that was after we had worked around the problems they created. Using
async replication simply meant that we were accepting the replication lag
window as a period of time where a revocation event might not apply. I
don't know that we ever got hard numbers, but with the write data we had
we speculated the worst result would be that you'd revoke a token in
Dallas and Sydney or London might keep working with that token for
however long a latency storm lasted + the recovery time for applying the
backlog. At worst, a few minutes.

Either way, it should be much simpler to manage slave lag than to deal
with a Galera cluster that won't accept any writes at all because it
can't get quorum.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Keystone] Cockroachdb for Keystone Multi-master

[Clint Byrum]

Excerpts from Jay Pipes's message of 2017-05-30 14:52:01 -0400:
> Sorry for the delay in getting back on this... comments inline.
> 
> On 05/18/2017 06:13 PM, Adrian Turjak wrote:
> > Hello fellow OpenStackers,
> > 
> > For the last while I've been looking at options for multi-region
> > multi-master Keystone, as well as multi-master for other services I've
> > been developing and one thing that always came up was there aren't many
> > truly good options for a true multi-master backend.
> 
> Not sure whether you've looked into Galera? We had a geo-distributed 
> 12-site Galera cluster servicing our Keystone assignment/identity 
> information WAN-replicated. Worked a charm for us at AT&T. Much easier 
> to administer than master-slave replication topologies and the 
> performance (yes, even over WAN links) of the ws-rep replication was 
> excellent. And yes, I'm aware Galera doesn't have complete snapshot 
> isolation support, but for Keystone's workloads (heavy, heavy read, very 
> little write) it is indeed ideal.
> 

This has not been my experience.

We had a 3 site, 9 node global cluster and it was _extremely_ sensitive
to latency. We'd lose even read ability whenever we had a latency storm
due to quorum problems.

Our sites were London, Dallas, and Sydney, so it was pretty common for
there to be latency between any of them.

I lost track of it after some reorgs, but I believe the solution was
to just have a single site 3-node galera for writes, and then use async
replication for reads. We even helped land patches in Keystone to allow
split read/write host configuration.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Is the pendulum swinging on PaaS layers?

[Clint Byrum]

Excerpts from Clark Boylan's message of 2017-05-19 10:03:23 -0700:
> On Fri, May 19, 2017, at 05:59 AM, Duncan Thomas wrote:
> > On 19 May 2017 at 12:24, Sean Dague  wrote:
> > 
> > > I do get the concerns of extra logic in Nova, but the decision to break
> > > up the working compute with network and storage problem space across 3
> > > services and APIs doesn't mean we shouldn't still make it easy to
> > > express some pretty basic and common intents.
> > 
> > Given that we've similar needs for retries and race avoidance in and
> > between glance, nova, cinder and neutron, and a need or orchestrate
> > between at least these three (arguably other infrastructure projects
> > too, I'm not trying to get into specifics), maybe the answer is to put
> > that logic in a new service, that talks to those four, and provides a
> > nice simple API, while allowing the cinder, nova etc APIs to remove
> > things like internal retries?
> 
> The big issue with trying to solve the problem this way is that various
> clouds won't deploy this service then your users are stuck with the
> "base" APIs anyway or deploying this service themselves. This is mostly
> ok until you realize that we rarely build services to run "on" cloud
> rather than "in" cloud so I as the user can't sanely deploy a new
> service this way, and even if I can I'm stuck deploying it for the 6
> clouds and 15 regions (numbers not exact) because even more rarely do we
> write software that is multicloud/region aware.
> 
> We need to be very careful if this is the path we take because it often
> doesn't actually make the user experience better.

I think an argument can be made that if the community were to rally
around something like Enamel and Oaktree, that it would be deployed
broadly.

As Zane pointed out elsewhere in the thread, Heat does some of this
for you, and has seen a lot of adoption, but nowhere near the level
of Neutron and Cinder. However, I believe Heat is missing from some
clouds because it is stateful, and thus, requires a large investment to
deploy. Oaktree is specifically _not_ stateful, and not dependent on admin
access to function, so I could see rallying around something that _can_
be deployed by users, but would be much more popular for deployers to
add in as users ask for it.

So whatever gets chosen as the popular porcelein API, it sounds to me
like it's worth getting serious about it.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images?

[Clint Byrum]

Excerpts from Michał Jastrzębski's message of 2017-05-15 10:52:12 -0700:
> > Container images introduce some extra complexity, over the basic
> > operating system style packages mentioned above. Due to the way
> > they are constructed, they are likely to include content we don't
> > produce ourselves (either in the form of base layers or via including
> > build tools or other things needed when assembling the full image).
> > That extra content means there would need to be more tracking of
> > upstream issues (bugs, CVEs, etc.) to ensure the images are updated
> > as needed.
> 
> We can do this by building daily, which was the plan in fact. If we
> build every day you have at most 24hrs old packages, CVEs and things
> like that on non-openstack packages are still maintained by distro
> maintainers.
> 

What's at stake isn't so much "how do we get the bits to the users" but
"how do we only get bits to users that they need". If you build and push
daily, do you expect all of your users to also _pull_ daily? Redeploy
all their containers? How do you detect that there's new CVE-fixing
stuff in a daily build?

This is really the realm of distributors that have full-time security
teams tracking issues and providing support to paying customers.

So I think this is a fine idea, however, it needs to include a commitment
for a full-time paid security team who weighs in on every change to
the manifest. Otherwise we're just lobbing time bombs into our users'
data-centers.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][concurrency] lockutils lock fairness / starvation

[Clint Byrum]

Excerpts from Ben Nemec's message of 2017-05-15 15:48:33 -0500:
> 
> On 05/15/2017 03:24 PM, Doug Hellmann wrote:
> > Excerpts from Legacy, Allain's message of 2017-05-15 19:20:46 +:
> >>> -Original Message-
> >>> From: Doug Hellmann [mailto:d...@doughellmann.com]
> >>> Sent: Monday, May 15, 2017 2:55 PM
> >> <...>
> >>>
> >>> Excerpts from Legacy, Allain's message of 2017-05-15 18:35:58 +:
>  import eventlet
>  eventlet.monkey_patch
> >>>
> >>> That's not calling monkey_patch -- there are no '()'. Is that a typo?
> >>
> >> Yes, sorry, that was a typo when I put it in to the email.  It did have ()
> >> at the end.
> >>
> >>>
> >>> lock() claims to work differently when monkey_patch() has been called.
> >>> Without doing the monkey patching, I would expect the thread to have to
> >>> explicitly yield control.
> >>>
> >>> Did you see the problem you describe in production code, or just in this
> >>> sample program?
> >>
> >> We see this in production code.   I included the example to boil this down 
> >> to
> >> a simple enough scenario to be understood in this forum without the
> >> distraction of superfluous code.
> >>
> >
> > OK. I think from the Oslo team's perspective, this is likely to be
> > considered a bug in the application. The concurrency library is not
> > aware that it is running in an eventlet thread, so it relies on the
> > application to call the monkey patching function to inject the right
> > sort of lock class.  If that was done in the wrong order, or not
> > at all, that would cause this issue.
> 
> Does oslo.concurrency make any fairness promises?  I don't recall that 
> it does, so it's not clear to me that this is a bug.  I thought fair 
> locking was one of the motivations behind the DLM discussion.  My view 
> of the oslo.concurrency locking was that it is solely concerned with 
> preventing concurrent access to a resource.  There's no queuing or 
> anything that would ensure a given consumer can't grab the same lock 
> repeatedly.
> 

DLM is more about fairness between machines, not threads.

However, I'd agree that oslo.concurrency isn't making fairness
guarantees. It does claim to return a threading.Semaphore or
semaphore.Semaphore, neither of which facilitate fairness (nor would a
full fledged mutex).

In order to implement fairness you'll need every lock request to happen
in a FIFO queue. This is often implemented with a mutex-protected queue
of condition variables. Since the mutex for the queue is only held while
you append to the queue, you will always get the items from the queue
in the order they were written to it.

So you have lockers add themselves to the queue and wait on their
condition variable, and then a thread running all the time that reads
the queue and acts on each condition to make sure only one thread is
activated at a time (or that one thread can just always do all the work
if the arguments are simple enough to put in a queue).

> I'm also not really surprised that this example serializes all the 
> workers.  The operation being performed in each thread is simple enough 
> that it probably completes before a context switch could reasonably 
> occur, greenthreads or not.  Unfortunately one of the hard parts of 
> concurrency is that the "extraneous" details of a use case can end up 
> being important.
> 

It also gets hardware sensitive when you have true multi-threading,
since a user on a 2 core box will see different results than a 4 core.

> >
> > The next step is to look at which application had the problem, and under
> > what circumstances. Can you provide more detail there?
> 
> +1, although as I noted above I'm not sure this is actually a "bug".  It 
> would be interesting to know what real world use case is causing a 
> pathologically bad lock pattern though.
> 

I think it makes sense, especially in the greenthread example where
you're immediately seeing activity on the recently active socket and
thus just stay in that greenthread.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [architecture] Arch-WG, we hardly knew ye..

[Clint Byrum]

Excerpts from lebre.adrien's message of 2017-04-07 15:50:05 +0200:
> Dear Clint, Dear all, 
> 
> It is indeed unfortunate that the WG ends.
> 
> From our side (Inria folks from the Discovery initiative [1]), we had planned 
> to join the effort after the Boston Summit as we are currently addressing two 
> points that are/were closely related to the Arch WG (at least from our 
> understanding): 
> 
> * We have been working during this last cycle on the consolidation of the 
> EnOS solution [2] (in particular with advice from the performance team). 
> EnOS aims to perform OpenStack Performance analyses/profiling.  It is built 
> on top of Kolla and integrates Rally and Shaker and more recently  OSProfiler.
> We are currently conducting preliminary experiments to draw, in an automatic 
> manner, sequence diagrams such as 
> https://docs.openstack.org/ops-guide/_images/provision-an-instance.png
> We hope it will help our community to understand better the OpenStack 
> architecture as well as the interactions between the different services, in 
> particular it would help us to follow changes between cycles. 
> 
> * We started a discussion regarding the oslo.messaging driver [3]. 
> Our goal is, first,  to make a qualitative analysis of AMQP messages (i.e. we 
> would like to understand the different AMQP exchanges better) and try to 
> identify possible improvements. 
> Second, we will do performance analysis of the rabbitMQ under different 
> scenarios and, according to gathered results, we will conduct additional 
> experiments with alternatives solutions (ZMQ, Qpid, ...)
> Please note that this is a preliminary discussion, so we just exchanged 
> between a few folks from the Massively Distributed WG. We proposed a session 
> to the forum [4] with the goal of opening the discussion more generally to 
> the community. 
> 
> We are convinced of the relevance of a WG such as the Architecture one, but 
> as you probably already know, it is difficult for most of us to join 
> different meetings several times per week, and thus rather difficult to 
> gather efforts that are done in other WGs. I don't know whether and how we 
> can improve this solution but it would make sense. 
> 

Hi! I highly recommend working with the performance team for both of
those efforts:

https://wiki.openstack.org/wiki/Meetings/Performance

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [architecture] Arch-WG, we hardly knew ye..

[Clint Byrum]

I'm going to be blunt. I'm folding the Architecture Working Group
immediately following our meeting today at 2000 UTC. We'll be using the
time to discuss continuity of the base-services proposal, and any other
draw-down necessary. After that our meetings will cease.

I had high hopes for the arch-wg, with so many joining us to discuss
things in Atlanta. But ultimately, we remain a very small group with
very limited resources, and so I don't think it's the best use of our
time to continue chasing the arch-wg.

Thanks everyone for your contributions. See you in the trenches.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kubernetes][go] External OpenStack Cloud Provider for Kubernetes

[Clint Byrum]

Excerpts from Chris Hoge's message of 2017-04-04 17:09:11 -0400:
> 
> > On Apr 2, 2017, at 4:29 PM, Monty Taylor  wrote:
> > 
> > On 03/29/2017 03:39 PM, Steve Gordon wrote:
> >> - Original Message -
> >>> From: "Davanum Srinivas" 
> >>> To: "Chris Hoge" 
> >>> Cc: "OpenStack Development Mailing List (not for usage questions)" 
> >>> ,
> >>> "kubernetes-sig-openstack" 
> >>> Sent: Wednesday, March 29, 2017 2:28:29 PM
> >>> Subject: Re: [openstack-dev] [kubernetes][go] External OpenStack Cloud 
> >>> Provider for Kubernetes
> >>> 
> >>> Team,
> >>> 
> >>> Repo is ready:
> >>> http://git.openstack.org/cgit/openstack/k8s-cloud-provider
> >>> 
> >>> I've taken the liberty of updating it with the latest changes in the
> >>> kubernetes/kubernetes repo:
> >>> https://review.openstack.org/#/q/project:openstack/k8s-cloud-provider is
> >>> ready
> >>> 
> >>> So logical next step would be to add CI jobs to test in OpenStack
> >>> Infra. Anyone interested?
> >> 
> >> One question I have around this - do we have a shared view of what the 
> >> ideal matrix of tested combinations would like? E.g. kubernetes master on 
> >> openstack project's master, kubernetes master on openstack project's 
> >> stable branches (where available), do we also need/want to test kubernetes 
> >> stable milestones, etc.
> >> 
> >> At a high level my goal would be the same as Chris's "k8s gating on 
> >> OpenStack in the same ways that it does on AWS and GCE." which would imply 
> >> reporting results on PRs proposed to K8S master *before* they merge but 
> >> not sure we all agree on what that actually means testing against in 
> >> practice on the OpenStack side of the equation?
> > 
> > I think we want to have jobs that have the ability to test:
> > 
> > 1) A proposed change to k8s-openstack-provider against current master of
> > OpenStack
> > 2) A proposed change to k8s-openstack-provider against a stable release
> > of OpenStack
> > 3) A proposed change to OpenStack against current master of
> > k8s-openstack-provider
> > 4) A proposed change to OpenStack against stable release of
> > k8s-openstack-provider
> > 
> > Those are all easy now that the code is in gerrit, and it's well defined
> > what triggers and where it reports.
> > 
> > Additionally, we need to test the surface area between
> > k8s-openstack-provider and k8s itself. (if we wind up needing to test
> > k8s against proposed changes to OpenStack then we've likely done
> > something wrong in life)
> > 
> > 5) A proposed change to k8s-openstack-provider against current master of k8s
> > 6) A proposed change to k8s-openstack-provider against a stable release
> > of k8s
> > 7) A proposed change to k8s against current master of k8s-openstack-provider
> > 8) A proposed change to k8s against stable release of k8s-openstack-provider
> > 
> > 5 and 6 are things we can do right now. 7 and 8 will have to wait for GH
> > support to land in zuul (without which we can neither trigger test jobs
> > on proposed changes to k8s nor can we report the results back to anyone)
> 
> 7 and 8 are going to be pretty important for integrating into the K8S
> release process. At the risk of having a work item thrown at me,
> is there a target for when that feature will land?
> 

Hi! Github support is happening basically as "zuulv3+1". We're working
on it in parallel with the v3 effort, so it should be a relatively quick
+1, but I'd expect infra will need a couple months of shaking out v3
bugs and getting everything ported before we can start talking about
hooking infra's zuul up to Github.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][requirements][designate][cinder] Reverting eventlet version bump

[Clint Byrum]

Excerpts from Monty Taylor's message of 2017-04-04 12:19:36 -0500:
> On 04/04/2017 09:19 AM, Jay S Bryant wrote:
> > Monty,
> >
> > I agree with your approach.  Think we should not break other projects
> > with a change like this and the answer thus far has been to just patch
> > each project individually to work around the issues introduced by
> > eventlet.  So, I support this approach and think Sean would as well.
> 
> To follow up with everyone - dims and I spent a bit of time this morning 
> testing combinations of things, and it seems that master of eventlet 
> actually also does not fix things - there are some issues that will just 
> need to be figured out.
> 
> The suggested path forward at the moment is to pull designate out of the 
> requirements-sync process so it can stay pinned at 0.19 while the issues 
> are sorted out. A patch to bump designate back to 0.20.1 can accompany 
> the patch to fix it for 0.20.1 - and can be done as people have time, 
> rather than in a rush.
> 

Has anyone made any attempt to eliminate eventlet from Designate? The
less places it's used, the less problems OpenStack seems to have, IMO.
But I can see on cursory examination it has quite a few services so
perhaps it's just too entrenched?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Clint Byrum]

Excerpts from Sean Dague's message of 2017-03-15 08:54:55 -0400:
> On 03/15/2017 02:16 AM, Clint Byrum wrote:
> > Excerpts from Monty Taylor's message of 2017-03-15 04:36:24 +0100:
> >> On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
> >>> Team,
> >>>
> >>> So one more thing popped up again on IRC:
> >>> https://etherpad.openstack.org/p/oslo.config_etcd_backend
> >>>
> >>> What do you think? interested in this work?
> >>>
> >>> Thanks,
> >>> Dims
> >>>
> >>> PS: Between this thread and the other one about Tooz/DLM and
> >>> os-lively, we can probably make a good case to add etcd as a base
> >>> always-on service.
> >>
> >> As I mentioned in the other thread, there was specific and strong
> >> anti-etcd sentiment in Tokyo which is why we decided to use an
> >> abstraction. I continue to be in favor of us having one known service in
> >> this space, but I do think that it's important to revisit that decision
> >> fully and in context of the concerns that were raised when we tried to
> >> pick one last time.
> >>
> >> It's worth noting that there is nothing particularly etcd-ish about
> >> storing config that couldn't also be done with zk and thus just be an
> >> additional api call or two added to Tooz with etcd and zk drivers for it.
> >>
> > 
> > Combine that thought with the "please have an ingest/export" thought,
> > and I think you have a pretty operator-friendly transition path. Would
> > be pretty great to have a release of OpenStack that just lets you add
> > an '[etcd]', or '[config-service]' section maybe, to your config files,
> > and then once you've fully migrated everything, lets you delete all the
> > other sections. Then the admin nodes still have the full configs and
> > one can just edit configs in git and roll them out by ingesting.
> > 
> > (Then the magical rainbow fairy ponies teach our services to watch their
> > config service for changes and restart themselves).
> 
> Make sure to add:
> 
> ... (after fully quiescing, when they are not processing any inflight
> work, when they are part of a pool so that they can be rolling restarted
> without impacting other services trying to connect to them, with a
> rollback to past config should the new config cause a crash).
> 
> There are a ton of really interesting things about a network registry,
> that makes many things easier. However, from an operational point of
> view I would be concerned about the idea of services restarting
> themselves in a non orchestrated manner. Or that a single key set in the
> registry triggers a complete reboot of the cluster. It's definitely less
> clear to understand the linkage of the action that took down your cloud
> and why when the operator isn't explicit about "and restart this service
> now".
> 

It's big and powerful and scary. That's for sure. But it's not _that_
different than an Ansible playbook in its ability to massively complicate
or uncomplicate your life with a tiny amount of code. :)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Clint Byrum]

Excerpts from Doug Hellmann's message of 2017-03-15 15:35:13 -0400:
> Excerpts from Thomas Herve's message of 2017-03-15 09:41:16 +0100:
> > On Wed, Mar 15, 2017 at 12:05 AM, Joshua Harlow  
> > wrote:
> > 
> > > * How does reloading work (does it)?
> > 
> > No. There is nothing that we can do in oslo that will make services
> > magically reload configuration. It's also unclear to me if that's
> > something to do. In a containerized environment, wouldn't it be
> > simpler to deploy new services? Otherwise, supporting signal based
> > reload as we do today should be trivial.
> 
> Reloading works today with files, that's why the question is important
> to think through. There is a special flag to set on options that are
> "mutable" and then there are functions within oslo.config to reload.
> Those are usually triggered when a service gets a SIGHUP or something similar.
> 
> We need to decide what happens to a service's config when that API
> is used and the backend is etcd. Maybe nothing, because every time
> any config option is accessed the read goes all the way through to
> etcd? Maybe a warning is logged because we don't support reloads?
> Maybe an error is logged? Or maybe we flush the local cache and start
> reading from etcd on future accesses?
> 

etcd provides the ability to "watch" keys. So one would start a thread
that just watches the keys you want to reload on, and when they change
that thread will see a response and can reload appropriately.

https://coreos.com/etcd/docs/latest/dev-guide/api_reference_v3.html

see "service Watch"

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [infra][security] Encryption in Zuul v3

[Clint Byrum]

Excerpts from corvus's message of 2017-03-21 09:36:41 -0700:
> Hi,
> 
> In working on the implementation of the encrypted secrets feature of
> Zuul v3, I have found some things that warrant further discussion.  It's
> important to be deliberate about this and I welcome any feedback.
> 

Thanks for looking into this deeply.

> For reference, here is the relevant portion of the Zuul v3 spec:
> 
> http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html#secrets
> 
> And here is an implementation of that:
> 
> https://review.openstack.org/#/q/status:open+topic:secrets+project:openstack-infra/zuul
> 
> The short version is that we want to allow users to store private keys
> in the public git repos which Zuul uses to run jobs.  To do this, we
> propose to use asymmetric cryptography (RSA) to encrypt the data.  The
> specification suggests implementing PKCS#1-OAEP, a standard for
> implementing RSA encryption.
> 
> Note that RSA is not able to encrypt a message longer than the key, and
> PKCS#1 includes some overhead which eats into that.  If we use 4096 bit
> RSA keys in Zuul, we will be able to encrypt 3760 bits (or 470 bytes) of
> information.
> 

Hm, I must have read the standard wrong, I thought it was 480 bytes. I
very much trust your reading of it and experimentation with it above my
skimming.

> Further, note that value only holds if we use SHA-1.  It has been
> suggested that we may want to consider using SHA-256 with PKCS#1.  If we
> do, we will be able to encrypt slightly less data.  However, I'm not
> sure that the Python cryptography library allows this (yet?).  Also, see
> this answer for why it may not be necessary to use SHA-256 (and also,
> why we may want to anyway):
> 
> https://security.stackexchange.com/questions/112029/should-sha-1-be-used-with-rsa-oaep
> 

I think our hand will get forced into SHA256, and if we can land SHA256
support in cryptography's PKCS#1, we should. But until that's done, SHA1
is cryptographically sound, and I think we don't have time to get things
into perfect shape, so as long as we can accommodate SHA256 when it's
ready, seems like the mathemeticians think SHA1 is fine in PKCS#1. 

> One thing to note is that the OpenSSL CLI utility uses SHA-1.  Right
> now, I have a utility script which uses that to encrypt secrets so that
> it's easy for anyone to encrypt a secret without installing many
> dependencies.  Switching to another hash function would probably mean we
> wouldn't be able to use that anymore.  But that's also true for other
> systems (see below).
> 

We may just have to require python and a recent cryptography if people
are unable to use a SHA1 PKCS#1.

> In short, PKCS#1 pros: Simple, nicely packaged asymmetric encryption,
> hides plaintext message length (up to its limit).  Cons: limited to 470
> bytes (or less).
> 

I'd list the confusion around SHA1 as a con as well.

> Generally, when faced with the prospect of encrypting longer messages,
> the advice is to adopt a hybrid encryption scheme (as opposed to, say,
> chaining RSA messages together, or increasing the RSA key size) which
> uses symmetric encryption with a single-use key for the message and
> asymmetric encryption to hide the key.  If we want Zuul to support the
> encryption of longer secrets, we may want to adopt the hybrid approach.
> A frequent hybrid approach is to encrypt the message with AES, and then
> encrypt the AES key with RSA.
> 
> The hiera-eyaml work which originally inspired some of this is based on
> PKCS#7 with AES as the cipher -- ultimately a hybrid approach.  An
> interesting aspect of that implementation is that the use of PKCS#7 as a
> message passing format allows for multiple possible underlying ciphers
> since the message is wrapped in ASN.1 and is self-descriptive.  We might
> have simply chosen to go with that except that there don't seem to be
> many good options for implementing this in Python, largely because of
> the nightmare that is ASN.1 parsing.
> 
> The system we have devised for including encrypted content in our YAML
> files involves a YAML tag which specifies the encryption scheme.  So we
> can evolve our use to add or remove systems as needed in the future.
> 
> So to break this down into a series of actionable questions:
> 
> 1) Do we want a system to support encrypting longer secrets?  Our PKCS#1
> system supports up to 470 bytes.  That should be sufficient for most
> passwords and API keys, but unlikely to be sufficient for some
> certificate related systems, etc.
> 

Ultimately yes. The issue brought up earlier about SSH keys suggests
we'll likely need to be able to handle private keys that are 4096 bits
long.

> 2) If so, what system should we use?
> 
>2.1a) GPG?  This has hybrid encryption and transport combined.
>Implementation is likely to be a bit awkward, probably involving
>popen to external processes.
> 

As awkward as popening the cli's can be, there are at least libraries
for it:

http://pythonhosted.org/gnupg/

Re: [openstack-dev] [infra][security] Encryption in Zuul v3

[Clint Byrum]

Excerpts from Matthieu Huin's message of 2017-03-21 18:43:49 +0100:
> Hello James,
> 
> Thanks for opening the discussion on this topic. I'd like to mention that a
> very common type of secrets that are used in Continuous Deployments
> scenarios are SSH keys. Correct me if I am wrong, but PKCS#1 wouldn't
> qualify if standard keys were to be stored.

You could store a key, just not a 4096 bit key.

PKCS#1 has a header/padding of something like 12 bytes, and then you
need a hash in there, so for SHA1 that's 160 bits or 20 bytes, SHA256
is 256 bites so 32 bytes. So with a 4096 bit (512 bytes) Zuul key, you
can encrypt 480 bytes of plaintext, or 468 with sha256. That's enough
for a 3072 bit (384 bytes) SSH key. An uncommon size, but RSA says'
they're good past 2030:

https://www.emc.com/emc-plus/rsa-labs/historical/twirl-and-rsa-key-size.htm

It's a little cramped, but hey, this is the age of tiny houses, maybe we
should make do with what we have.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum][osc] What name to use for magnum commands in osc?

[Clint Byrum]

Excerpts from Adrian Otto's message of 2017-03-20 22:19:14 +:
> I was unsure, so I found him on IRC to clarify, and he pointed me to the 
> openstack/service-types-authority repository, where I submitted patch 445694 
> for review. We have three distinct identifiers in play:
> 
> 1) Our existing service catalog entry name: container-infra
> 2) Our openstack client noun: TBD, decision expected from our team tomorrow. 
> My suggestion: "coe cluster”.
> 3) Our (proposed) service type: coe-cluster
> 
> Each identifier has respective guidelines and limits, so they differ.
> 
> Adrian

Oh neat, I didn't even know that repository existed. TIL.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum][osc] What name to use for magnum commands in osc?

[Clint Byrum]

Excerpts from Adrian Otto's message of 2017-03-20 21:16:09 +:
> Jay,
> 
> On Mar 20, 2017, at 12:35 PM, Jay Pipes 
> mailto:jaypi...@gmail.com>> wrote:
> 
> On 03/20/2017 03:08 PM, Adrian Otto wrote:
> Team,
> 
> Stephen Watson has been working on an magnum feature to add magnum commands 
> to the openstack client by implementing a plugin:
> 
> https://review.openstack.org/#/q/status:open+project:openstack/python-magnumclient+osc
> 
> In review of this work, a question has resurfaced, as to what the client 
> command name should be for magnum related commands. Naturally, we’d like to 
> have the name “cluster” but that word is already in use by Senlin.
> 
> Unfortunately, the Senlin API uses a whole bunch of generic terms as 
> top-level REST resources, including "cluster", "event", "action", "profile", 
> "policy", and "node". :( I've warned before that use of these generic terms 
> in OpenStack APIs without a central group responsible for curating the API 
> would lead to problems like this. This is why, IMHO, we need the API working 
> group to be ultimately responsible for preventing this type of thing from 
> happening. Otherwise, there ends up being a whole bunch of duplication and 
> same terms being used for entirely different things.
> 
> >Stephen opened a discussion with Dean Troyer about this, and found that 
> >“infra” might be a suitable name and began using that, and multiple team 
> >members are not satisfied with it.
> 
> Yeah, not sure about "infra". That is both too generic and not an actual 
> "thing" that Magnum provides.
> 
> > The name “magnum” was excluded from consideration because OSC aims to be 
> > project name agnostic. We know that no matter what word we pick, it’s not 
> > going to be ideal. I’ve added an agenda on our upcoming team meeting to 
> > judge community consensus about which alternative we should select:
> 
> https://wiki.openstack.org/wiki/Meetings/Containers#Agenda_for_2017-03-21_1600_UTC
> 
> Current choices on the table are:
> 
>  * c_cluster (possible abbreviation alias for container_infra_cluster)
>  * coe_cluster
>  * mcluster
>  * infra
> 
> For example, our selected name would appear in “openstack …” commands. Such 
> as:
> 
> $ openstack c_cluster create …
> 
> If you have input to share, I encourage you to reply to this thread, or come 
> to the team meeting so we can consider your input before the team makes a 
> selection.
> 
> What is Magnum's service-types-authority service_type?
> 
> I propose "coe-cluster” for that, but that should be discussed further, as 
> it’s impossible for magnum to conform with all the requirements for service 
> types because they fundamentally conflict with each other:
> 
> https://review.openstack.org/447694
> 
> In the past we referred to this type as a “bay” but found it burdensome for 
> users and operators to use that term when literally bay == cluster. We just 
> needed to call it what it is because there’s a prevailing name for that 
> concept, and everyone expects that’s what it’s called.

I Think Jay was asking for Magnum's name in the catalog:

Which is 'container-infra' according to this:

https://github.com/openstack/python-magnumclient/blob/master/magnumclient/v1/client.py#L34

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [dib][heat] dib-utils/dib-run-parts/dib v2 concern

[Clint Byrum]

Excerpts from Clark Boylan's message of 2017-03-16 10:16:37 -0700:
> On Thu, Mar 16, 2017, at 09:46 AM, Steven Hardy wrote:
> > On Thu, Mar 16, 2017 at 10:30:48AM -0500, Gregory Haynes wrote:
> > > On Thu, Mar 16, 2017, at 05:18 AM, Steven Hardy wrote:
> > > > On Wed, Mar 15, 2017 at 04:22:37PM -0500, Ben Nemec wrote:
> > > > > While looking through the dib v2 changes after the feature branch was 
> > > > > merged
> > > > > to master, I noticed this commit[1], which bring dib-run-parts back 
> > > > > into dib
> > > > > itself.  Unfortunately I missed the original proposal to do this, but 
> > > > > I have
> > > > > some concerns about the impact of this change.
> > > > > 
> > > > > Originally the split was done so that dib-run-parts and one of the
> > > > > os-*-config projects (looks like os-refresh-config) that depends on 
> > > > > it could
> > > > > be included in a stock distro cloud image without pulling in all of 
> > > > > dib.
> > > > > Note that it is still present in the requirements of orc: 
> > > > > https://github.com/openstack/os-refresh-config/blob/master/requirements.txt#L5
> > > > > 
> > > > > Disk space in a distro cloud image is at a premium, so pulling in a 
> > > > > project
> > > > > like diskimage-builder to get one script out of it was not 
> > > > > acceptable, at
> > > > > least from what I was told at the time.
> > > > > 
> > > > > I believe this was done so a distro cloud image could be used with 
> > > > > Heat out
> > > > > of the box, hence the heat tag on this message.  I don't know exactly 
> > > > > what
> > > > > happened after we split out dib-utils, so I'm hoping someone can 
> > > > > confirm
> > > > > whether this requirement still exists.  I think Steve was the one who 
> > > > > made
> > > > > the original request.  There were a lot of Steves working on Heat at 
> > > > > the
> > > > > time though, so it's possible I'm wrong. ;-)
> > > > 
> > > > I don't think I'm the Steve you're referring to, but I do have some
> > > > additional info as a result of investigating this bug:
> > > > 
> > > > https://bugs.launchpad.net/tripleo/+bug/1673144
> > > > 
> > > > It appears we have three different versions of dib-run-parts on the
> > > > undercloud (and, presumably overcloud nodes) at the moment, which is a
> > > > pretty major headache from a maintenance/debugging perspective.
> > > > 
> > > 
> > > I looked at the bug and I think there may only be two different
> > > versions? The versions in /bin and /usr/bin seem to come from the same
> > > package (so I hope they are the same version). I don't understand what
> > > is going on with the ./lib version but that seems like either a local
> > > package / checkout or something else non-dib related.
> > > 
> > > Two versions is certainly less than ideal, though :).
> > 
> > No I think there are four versions, three unique:
> > 
> > (undercloud) [stack@undercloud ~]$ rpm -qf /usr/bin/dib-run-parts
> > dib-utils-0.0.11-1.el7.noarch
> > (undercloud) [stack@undercloud ~]$ rpm -qf /bin/dib-run-parts
> > dib-utils-0.0.11-1.el7.noarch
> > (undercloud) [stack@undercloud ~]$ rpm -qf
> > /usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts
> > diskimage-builder-2.0.1-0.20170314023517.756923c.el7.centos.noarch
> > (undercloud) [stack@undercloud ~]$ rpm -qf /usr/local/bin/dib-run-parts
> > file /usr/local/bin/dib-run-parts is not owned by any package
> > 
> > /usr/bin/dib-run-parts and /bin/dib-run-parts are the same file, owned by
> > dib-utils
> > 
> > /usr/lib/python2.7/site-packages/diskimage_builder/lib/dib-run-parts is
> > owned by diskimage-builder
> > 
> > /usr/local/bin/dib-run-parts is the mystery file presumed from image
> > building
> > 
> > But the exciting thing from a rolling-out-bugfixes perspective is that
> > the
> > one actually running via o-r-c isn't either of the packaged versions
> > (doh!)
> > so we probably need to track down which element is installing it.
> > 
> > This is a little OT for this thread (sorry), but hopefully provides more
> > context around my concerns about creating another fork etc.
> > 
> > > > However we resolve this, *please* can we avoid permanently forking the
> > > > tool, as e.g in that bug, where do I send the patch to fix leaking
> > > > profiledir directories?  What package needs an update?  What is
> > > > installing
> > > > the script being run that's not owned by any package?
> > > > 
> > > > Yes, I know the answer to some of those questions, but I'm trying to
> > > > point
> > > > out duplicating this script and shipping it from multiple repos/packages
> > > > is
> > > > pretty horrible from a maintenance perspective, especially for new or
> > > > casual contributors.
> > > > 
> > > 
> > > I agree. You answered my previous question of whether os-refresh-config
> > > is still in use (sounds like it definitely is) so this complicates
> > > things a bit.
> > > 
> > > > If we have to fork it, I'd suggest we should rename the script to avoid
> > > > the
> > > > confusio

Re: [openstack-dev] [all][ironic] Kubernetes-based long running processes

[Clint Byrum]

Excerpts from Dean Troyer's message of 2017-03-16 12:19:36 -0500:
> On Wed, Mar 15, 2017 at 5:28 PM, Taryma, Joanna  
> wrote:
> > I’m reaching out to you to ask if you’re aware of any other use cases that
> > could leverage such solution. If there’s a need for it in other project, it
> > may be a good idea to implement this in some sort of a common place.
> 
> Before implementing something new it would be a good exercise to have
> a look at the other existing ways to run VMs and containers already in
> the OpenStack ecosystem.  Service VMs are a thing, and projects like
> Octavia are built around running inside the existing infrastructure.
> There are a bunch of deployment projects that are also designed
> specifically to run services with minimal base requirements.

The console access bit Joanna mentioned is special in that it needs to be
able to reach things like IPMI controllers. So that's not going to really
be able to run on a service VM easily. It's totally doable (I think we
could have achieved it with VTEP switches and OVN when I was playing
with that), but I can understand why a container solution running on
the same host as the conductor might be more desirable than service VMs.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Jay Pipes's message of 2017-03-15 15:06:26 -0400:
> +Boris B
> 
> On 03/15/2017 02:55 PM, Fox, Kevin M wrote:
> > I think they are. If they are not, things will break if federation is used 
> > for sure. If you know that it is please let me know. I want to deploy 
> > federation at some point but was waiting for dashboard support. Now that 
> > the dashboard supports it, I may try it soon. Its a no-go still though if 
> > heat doesn't work with it.
> 
> We had a customer engagement recently that had issues with Heat not 
> being able to execute certain actions in a federated Keystone 
> environment. I believe we learned that Keystone trusts and federation 
> were not compatible during this engagement.
> 
> Boris, would you mind refreshing memories on this?

Is it possible that this was because there was no writable domain for
Heat to create instance users in?

Because when last I used Heat long ago, Heat straight up just won't work
without trusts (since you have to give Heat a trust for it to be able
to do anything for you). Prior to that Heat was storing your creds in
its database... pretty sure that's long gone.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Clint Byrum]

Excerpts from Clark Boylan's message of 2017-03-15 09:50:49 -0700:
> On Wed, Mar 15, 2017, at 06:19 AM, Monty Taylor wrote:
> > On 03/15/2017 11:37 AM, Davanum Srinivas wrote:
> > > Monty, Team,
> > > 
> > > Sorry for the top post:
> > > 
> > > Support for etcd/tooz in devstack (with file driver as default) -
> > > https://review.openstack.org/#/c/445432/
> > > 
> > > As of right now both zookeeper driver and etcd driver is working fine:
> > > https://review.openstack.org/#/c/445630/
> > > https://review.openstack.org/#/c/445629/
> > > 
> > > The problem we have from before is that we do not have any CI jobs
> > > that used zookeeper.
> > > 
> > > I am leaning towards just throwing the etcd as default and if folks
> > > are interested in zookeeper then they can add specific CI jobs with
> > > DLM_BACKEND variable set.
> > 
> > That doesn't bother me - zk as the default choice was because at the
> > time zk worked and etcd did not.
> > 
> > That said - etcd3 is a newer/better thing - so maybe instead of driving
> > etcd home as a default before we add etcd3 support, we just change tooz
> > to support etcd3, add the devstack jobs to use that, and start from a
> > position that doesn't involve dealing with any legacy?
> > 
> 
> One logistical concern that no one else seems to have pointed out on
> this thread yet is that the example devstack setup linked at the
> beginning of the thread
> (http://git.openstack.org/cgit/openstack/dragonflow/tree/devstack/etcd_driver)
> grabs tarball from github to perform the etcd3 installation. Looks like
> Fedora and CentOS may have proper packages for etcd3 but Ubuntu does
> not.
> 
> For reliability reasons we probably do not want to be grabbing this
> tarball from github on every job run (particularly if this becomes a
> "base service" installed on every devstack job run).
> 

Zesty looks to be getting etcd3, though there are dependencies that
haven't synced/built yet[1].  Once that's done we can submit a backport and
get it available on Xenial and then just 'apt-get install backports/etcd'

Of course, using backports in LTS's is a little bit weird since the
security team won't maintain backports.

[1] https://launchpad.net/ubuntu/+source/etcd/3.1.0-1/+build/12016435

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Kristi Nikolla's message of 2017-03-15 12:35:45 -0400:
> This might be related to the current discussion. 
> 
> In one of the keystone PTG sessions we started to talk about API keys. [0]
> A spec is being written and discussed. [1]
> 
> This would allow the user to provision API key credentials with a subset of 
> roles for use inside of their applications. Removing the need to inject the 
> actual user credentials inside an application.
> 
> [0] http://lbragstad.com/keystone-pike-ptg-summary/
> [1] https://review.openstack.org/#/c/438761
> 

 ^^ this!

> > On Mar 15, 2017, at 8:45 AM, Sean Dague  wrote:
> > 
> > On 03/13/2017 05:10 PM, Zane Bitter wrote:
> >> 
> >> Demo please!
> >> 
> >> Most Keystone backends are read-only, you can't even create a new user
> >> account yourself. It's an admin-only API anyway. The only non-expiring
> >> credential you even *have*, ignoring the difficulties of getting it to
> >> the server, is your LDAP password. Would *you* put *your* LDAP password
> >> on an internet-facing server? I would not.
> > 
> > So is one of the issues to support cloud native flows that our user auth
> > system, which often needs to connect into traditional enterprise
> > systems, doesn't really consider that?
> > 
> > I definitely agree, if your cloud is using your LDAP password, which
> > gets you into your health insurance and direct deposit systems at your
> > employeer, sticking this into a cloud server is a no go.
> > 
> > Thinking aloud, I wonder if user provisionable sub users would help
> > here. They would have all the same rights as the main user (except
> > modify other subusers), but would have a dedicated user provisioned
> > password. You basically can carve off the same thing from Google when
> > you have services that can't do the entire oauth/2factor path. Fastmail
> > rolled out something similar recently as well.
> > 
> > -Sean
> > 
> > -- 
> > Sean Dague
> > http://dague.net
> > 
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Sean Dague's message of 2017-03-15 08:45:54 -0400:
> On 03/13/2017 05:10 PM, Zane Bitter wrote:
> 
> >> I'm not sure I agree. One can very simply inject needed credentials
> >> into a running VM and have it interact with the cloud APIs.
> > 
> > Demo please!
> > 
> > Most Keystone backends are read-only, you can't even create a new user
> > account yourself. It's an admin-only API anyway. The only non-expiring
> > credential you even *have*, ignoring the difficulties of getting it to
> > the server, is your LDAP password. Would *you* put *your* LDAP password
> > on an internet-facing server? I would not.
> 
> So is one of the issues to support cloud native flows that our user auth
> system, which often needs to connect into traditional enterprise
> systems, doesn't really consider that?
> 
> I definitely agree, if your cloud is using your LDAP password, which
> gets you into your health insurance and direct deposit systems at your
> employeer, sticking this into a cloud server is a no go.
> 
> Thinking aloud, I wonder if user provisionable sub users would help
> here. They would have all the same rights as the main user (except
> modify other subusers), but would have a dedicated user provisioned
> password. You basically can carve off the same thing from Google when
> you have services that can't do the entire oauth/2factor path. Fastmail
> rolled out something similar recently as well.
> 

Could we just let users manage a set of OAuth keys that have a subset
of their roles?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Clint Byrum]

Excerpts from Jay Pipes's message of 2017-03-14 22:13:32 -0400:
> On 03/14/2017 05:01 PM, Clint Byrum wrote:
> > Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
> >> On 03/14/2017 02:50 PM, Julien Danjou wrote:
> >>> On Tue, Mar 14 2017, Jay Pipes wrote:
> >>>
> >>>> Not tooz, because I'm not interested in a DLM nor leader election library
> >>>> (that's what the underlying etcd3 cluster handles for me), only a fast 
> >>>> service
> >>>> liveness/healthcheck system, but it shows usage of etcd3 and Google 
> >>>> Protocol
> >>>> Buffers implementing a simple API for liveness checking and host 
> >>>> maintenance
> >>>> reporting.
> >>>
> >>> Cool cool. So that's the same feature that we implemented in tooz 3
> >>> years ago. It's called "group membership". You create a group, make
> >>> nodes join it, and you know who's dead/alive and get notified when their
> >>> status change.
> >>
> >> The point of os-lively is not to provide a thin API over ZooKeeper's
> >> group membership interface. The point of os-lively is to remove the need
> >> to have a database (RDBMS) record of a service in Nova.
> >
> > That's also the point of tooz's group membership API:
> >
> > https://docs.openstack.org/developer/tooz/compatibility.html#grouping
> 
> Did you take a look at the code I wrote in os-lively? What part of the 
> tooz group membership API do you think I would have used?
> 
> Again, this was a weekend project that I was moving fast on. I looked at 
> tooz and didn't see how I could use it for my purposes, which was to 
> store a versioned object in a consistent key/value store with support 
> for transactional semantics when storing index and data records at the 
> same time [1]
> 
> https://github.com/jaypipes/os-lively/blob/master/os_lively/service.py#L468-L511
> 
> etcd3 -- and specifically etcd3, not etcd2 -- supports the transactional 
> semantics in a consistent key/value store that I needed.
> 

ZK has all the primitives necessary, and the client libs behind tooz for
it have transaction support basically identical to etcd3's:

https://kazoo.readthedocs.io/en/latest/_modules/kazoo/client.html#KazooClient.transaction

> tooz is cool, but it's not what I was looking for. It's solving a 
> different problem than I was trying to solve.
> 
> This isn't a case of NIH, despite what Julien is trying to intimate in 
> his emails.
> 

Yeah I don't want to imply that either. I'm trying to figure out how we
can add what you're doing to tooz, not why you didn't see something or
why you reinvented something.

> >> tooz simply abstracts a group membership API across a number of drivers.
> >> I don't need that. I need a way to maintain a service record (with
> >> maintenance period information, region, and an evolvable data record
> >> format) and query those service records in an RDBMS-like manner but
> >> without the RDBMS being involved.
> >>
> >>>> servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
> >>>> service liveness checking, which should dramatically reduce the amount 
> >>>> of both
> >>>> DB traffic as well as conductor/MQ service update traffic.
> >>>
> >>> Interesting. Joshua and Vilob tried to push usage of tooz group
> >>> membership a couple of years ago, but it got nowhere. Well, no, they got
> >>> 2 specs written IIRC:
> >>>
> >>>   
> >>> https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
> >>>
> >>> But then it died for whatever reasons on Nova side.
> >>
> >> It died because it didn't actually solve a problem.
> >>
> >> The problem is that even if we incorporate tooz, we would still need to
> >> have a service table in the RDBMS and continue to query it over and over
> >> again in the scheduler and API nodes.
> >
> > Most likely it was designed with hesitance to have a tooz requirement
> > to be a source of truth. But it's certainly not a problem for most tooz
> > backends to be a source of truth. Certainly not for etcd or ZK, which
> > are both designed to be that.
> >
> >> I want all service information in the same place, and I don't want to
> >> use an RDBMS for that information. etcd3 provides an ideal place to
> &g

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Clint Byrum]

Excerpts from Monty Taylor's message of 2017-03-15 04:36:24 +0100:
> On 03/14/2017 06:04 PM, Davanum Srinivas wrote:
> > Team,
> > 
> > So one more thing popped up again on IRC:
> > https://etherpad.openstack.org/p/oslo.config_etcd_backend
> > 
> > What do you think? interested in this work?
> > 
> > Thanks,
> > Dims
> > 
> > PS: Between this thread and the other one about Tooz/DLM and
> > os-lively, we can probably make a good case to add etcd as a base
> > always-on service.
> 
> As I mentioned in the other thread, there was specific and strong
> anti-etcd sentiment in Tokyo which is why we decided to use an
> abstraction. I continue to be in favor of us having one known service in
> this space, but I do think that it's important to revisit that decision
> fully and in context of the concerns that were raised when we tried to
> pick one last time.
> 
> It's worth noting that there is nothing particularly etcd-ish about
> storing config that couldn't also be done with zk and thus just be an
> additional api call or two added to Tooz with etcd and zk drivers for it.
> 

Combine that thought with the "please have an ingest/export" thought,
and I think you have a pretty operator-friendly transition path. Would
be pretty great to have a release of OpenStack that just lets you add
an '[etcd]', or '[config-service]' section maybe, to your config files,
and then once you've fully migrated everything, lets you delete all the
other sections. Then the admin nodes still have the full configs and
one can just edit configs in git and roll them out by ingesting.

(Then the magical rainbow fairy ponies teach our services to watch their
config service for changes and restart themselves).

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Clint Byrum]

Excerpts from Fox, Kevin M's message of 2017-03-15 00:10:24 +:
> +1 for having the option. I think in general it is a great idea.
> 
> I think there are some devils in the implementation. How do you prevent
> a service from getting way more secrets then it strictly needs? Maybe
> this is the start of an activity thought to split out all secrets from
> config, which would be an awesome thing to do. Having secrets commingled
> with config has always rubbed me the wrong way.
> 

You'll always have at least one secret around that grants access to
the others.  Whether it is the root SSH key, or the TLS private key that
grants etcd.

This is where we might diverge into the key management discussion so we
can at least have audit trails. With config only in files, you could
use something like CloudKeep (defunct?) that would effectively make
config-not-secrets into templates that would be filled with secrets when
read from a fifo backed by a daemon that fetched secrets from Barbican.

But, we could, instead, have less moving parts by having oslo.config
provide the option to just use castellan (or oslo.keymanager if we do
rename it) to handle key management of a TLS private key for etcd auth,
and maybe an additional private key for decryption of secrets like db
passwords and such. If you don't have Barbican, you can either a) write
a castellan driver for your existing key manager, b) help finish the
Hashicorp Vault castellan driver that Robert Clark was hacking on in
ATL, or c) use the insecure on-disk one that basically makes it work
just like your httpd /etc/httpd/ssl.key dir.

Whatever is done, there's a lot of operator-unfriendliness if it's not
accompanied by nice tools to read the effective configuration. Something
like `/opt/openstack/bin/keystone-manage show-config`.

> The other issue I can think of is overridability. the current
> separate file per instantiated service has some flexibility that a
> simple implementation of just looking in etcd for the keys may not work
> for. Some, look here first, then look for overrides over here thing
> would work though.
> 

That was addressed in the original I think. This would work:

/opt/openstack/bin/nova-conductor --config=/etc/nova/nova.conf 
--etcd-url=http://foo:94895/common --etcd-url=https://foo:94895/my-hostname

As with current oslo.config behavior, later urls would override earlier
ones. I don't think anybody's suggesting doing away with configs. Just
that having them in some central micro service means having less mutable
things to worry about in your images, which is why Kolla's asking so
nicely. :)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][barbican][castellan] Proposal to rename Castellan to oslo.keymanager

[Clint Byrum]

Excerpts from Doug Hellmann's message of 2017-03-14 20:05:54 -0400:
> Excerpts from Doug Hellmann's message of 2017-03-14 19:20:08 -0400:
> > Excerpts from Clint Byrum's message of 2017-03-13 13:49:22 -0700:
> > > Excerpts from Doug Hellmann's message of 2017-03-13 15:12:42 -0400:
> > > > Excerpts from Farr, Kaitlin M.'s message of 2017-03-13 18:55:18 +:
> > > > > Proposed library name: Rename Castellan to oslo.keymanager
> > > > > 
> > > > > Proposed library mission/motivation: Castellan's goal is to provide a
> > > > > generic key manager interface that projects can use for their key
> > > > > manager needs, e.g., storing certificates or generating keys for
> > > > > encrypting data.  The interface passes the commands and Keystone
> > > > > credentials on to the configured back end. Castellan is not a service
> > > > > and does not maintain state. The library can grow to have multiple
> > > > > back ends, as long as the back end can authenticate Keystone
> > > > > credentials.  The only two back end options now in Castellan are
> > > > > Barbican and a limited mock key manager useful only for unit tests.
> > > > > If someone wrote a Keystone auth plugin for Vault, we could also have 
> > > > > a
> > > > > Vault back end for Castellan.
> > > > > 
> > > > > The benefit of using Castellan versus using Barbican directly
> > > > > is Castellan allows the option of swapping out for other key managers,
> > > > > mainly for testing.  If projects want their own custom back end for
> > > > > Castellan, they can write a back end that implements the Castellan
> > > > > interface but lives in their own code base, i.e., ConfKeyManager in
> > > > > Nova and Cinder. Additionally, Castellan already has oslo.config
> > > > > options defined which are helpful for configuring the project to talk
> > > > > to Barbican.
> > > > > 
> > > > > When the Barbican team first created the Castellan library, we had
> > > > > reached out to oslo to see if we could name it oslo.keymanager, but 
> > > > > the
> > > > > idea was not accepted because the library didn't have enough traction.
> > > > > Now, Castellan is used in many projects, and we thought we would
> > > > > suggest renaming again.  At the PTG, the Barbican team met with the 
> > > > > AWG
> > > > > to discuss how we could get Barbican integrated with more projects, 
> > > > > and
> > > > > the rename was also suggested at that meeting.  Other projects are
> > > > > interested in creating encryption features, and a rename will help
> > > > > clarify the difference between Barbican and Castellan.
> > > > 
> > > > Can you expand on why you think that is so? I'm not disagreeing with the
> > > > statement, but it's not obviously true to me, either. I vaguely remember
> > > > having it explained at the PTG, but I don't remember the details.
> > > > 
> > > 
> > > To me, Oslo is a bunch of libraries that encompass "the way OpenStack
> > > does ". When  is key management, projects are, AFAICT, universally
> > > using Castellan at the moment. So I think it fits in Oslo conceptually.
> > > 
> > > As far as what benefit there is to renaming it, the biggest one is
> > > divesting Castellan of the controversy around Barbican. There's no
> > > disagreement that explicitly handling key management is necessary. There
> > > is, however, still hesitance to fully adopt Barbican in that role. In
> > > fact I heard about some alternatives to Barbican, namely "Vault"[1] and
> > > "Tang"[2], that may be useful for subsets of the community, or could
> > > even grow into de facto standards for key management.
> > > 
> > > So, given that there may be other backends, and the developers would
> > > like to embrace that, I see value in renaming. It would help, I think,
> > > Castellan's developers to be able to focus on key management and not
> > > have to explain to every potential user "no we're not Barbican's cousin,
> > > we're just an abstraction..".
> > > 
> > > > > Existing similar libraries (if any) and why they aren't being used: 
> > > > > N/A
> > > > > 
> > > > > Reviewer activity: Barbican team
> > > > 
> > > > If the review team is going to be largely the same, I'm not sure I
> > > > see the benefit of changing the ownership of the library. We certainly
> > > > have other examples of Oslo libraries being managed mainly by
> > > > sub-teams made up of folks who primarily focus on other projects.
> > > > oslo.policy and oslo.versionedobjects come to mind, but in both of
> > > > those cases the code was incubated in Oslo or brought into Oslo
> > > > before the tools for managing shared libraries were widely used
> > > > outside of the Oslo team. We now have quite a few examples of project
> > > > teams managing shared libraries (other than their clients).
> > > > 
> > > 
> > > While this makes sense, I'm not so sure any of those are actually
> > > specifically in the same category as Castellan. Perhaps you can expand
> > > on which libraries have done this, and how they're similar

Re: [openstack-dev] [oslo][kolla][openstack-helm][tripleo][all] Storing configuration options in etcd(?)

[Clint Byrum]

Excerpts from Davanum Srinivas's message of 2017-03-14 13:04:37 -0400:
> Team,
> 
> So one more thing popped up again on IRC:
> https://etherpad.openstack.org/p/oslo.config_etcd_backend
> 
> What do you think? interested in this work?
> 
> Thanks,
> Dims
> 
> PS: Between this thread and the other one about Tooz/DLM and
> os-lively, we can probably make a good case to add etcd as a base
> always-on service.
> 

This is a cool idea, and I think we should do it.

A few loose ends I'd like to see in a spec:

* Security Security Security. (Hoping if I say it 3 times a real
  security person will appear and ask the hard questions).
* Explain clearly how operators would inspect, edit, and diff their
  configs.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Clint Byrum]

Excerpts from Jay Pipes's message of 2017-03-14 15:30:32 -0400:
> On 03/14/2017 02:50 PM, Julien Danjou wrote:
> > On Tue, Mar 14 2017, Jay Pipes wrote:
> >
> >> Not tooz, because I'm not interested in a DLM nor leader election library
> >> (that's what the underlying etcd3 cluster handles for me), only a fast 
> >> service
> >> liveness/healthcheck system, but it shows usage of etcd3 and Google 
> >> Protocol
> >> Buffers implementing a simple API for liveness checking and host 
> >> maintenance
> >> reporting.
> >
> > Cool cool. So that's the same feature that we implemented in tooz 3
> > years ago. It's called "group membership". You create a group, make
> > nodes join it, and you know who's dead/alive and get notified when their
> > status change.
> 
> The point of os-lively is not to provide a thin API over ZooKeeper's 
> group membership interface. The point of os-lively is to remove the need 
> to have a database (RDBMS) record of a service in Nova.
> 

That's also the point of tooz's group membership API:

https://docs.openstack.org/developer/tooz/compatibility.html#grouping

> tooz simply abstracts a group membership API across a number of drivers. 
> I don't need that. I need a way to maintain a service record (with 
> maintenance period information, region, and an evolvable data record 
> format) and query those service records in an RDBMS-like manner but 
> without the RDBMS being involved.
> 
> >> servicegroup API with os-lively and eliminate Nova's use of an RDBMS for
> >> service liveness checking, which should dramatically reduce the amount of 
> >> both
> >> DB traffic as well as conductor/MQ service update traffic.
> >
> > Interesting. Joshua and Vilob tried to push usage of tooz group
> > membership a couple of years ago, but it got nowhere. Well, no, they got
> > 2 specs written IIRC:
> >
> >   
> > https://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/service-group-using-tooz.html
> >
> > But then it died for whatever reasons on Nova side.
> 
> It died because it didn't actually solve a problem.
> 
> The problem is that even if we incorporate tooz, we would still need to 
> have a service table in the RDBMS and continue to query it over and over 
> again in the scheduler and API nodes.
> 

Most likely it was designed with hesitance to have a tooz requirement
to be a source of truth. But it's certainly not a problem for most tooz
backends to be a source of truth. Certainly not for etcd or ZK, which
are both designed to be that.

> I want all service information in the same place, and I don't want to 
> use an RDBMS for that information. etcd3 provides an ideal place to 
> store service record information. Google Protocol Buffers is an ideal 
> data format for evolvable versioned objects. os-lively presents an API 
> that solves the problem I want to solve in Nova. tooz didn't.
> 

Was there something inherent in tooz's design that prevented you from
adding it to tooz's group API? Said API already includes liveness (watch
the group that corresponds to the service you want).

The only thing missing is being able to get groups and group members
by secondary indexes. etcd3's built in indexes by field are pretty nice
for that, but ZK can likely also do it too by maintaining the index in
the driver.

I understand abstractions can seem pretty cumbersome when you're moving
fast. It's not something I want to see stand in your way. But it would
be nice to see where there's deficiency in tooz so we can be there for
the next project that needs it and maybe eventually factor out direct
etcd3 usage so users who have maybe chosen ZK as their tooz backend can
also benefit from your work.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM

[Clint Byrum]

Excerpts from Fox, Kevin M's message of 2017-03-14 00:09:55 +:
> With my operator hat on, I would like to use the etcd backend, as I'm already 
> paying the cost of maintaining etcd clusters as part of Kubernetes. Adding 
> Zookeeper is a lot more work.
> 

It would probably be a good idea to put etcd in as a plugin and make
sure it is tested in the gate. IIRC nobody has requested the one thing
ZK can do that none of the others can (fair locking), but it's likely
there are bugs in both drivers since IIRC neither are enabled in any
gate.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][barbican][castellan] Proposal to rename Castellan to oslo.keymanager

[Clint Byrum]

Excerpts from Doug Hellmann's message of 2017-03-13 15:12:42 -0400:
> Excerpts from Farr, Kaitlin M.'s message of 2017-03-13 18:55:18 +:
> > Proposed library name: Rename Castellan to oslo.keymanager
> > 
> > Proposed library mission/motivation: Castellan's goal is to provide a
> > generic key manager interface that projects can use for their key
> > manager needs, e.g., storing certificates or generating keys for
> > encrypting data.  The interface passes the commands and Keystone
> > credentials on to the configured back end. Castellan is not a service
> > and does not maintain state. The library can grow to have multiple
> > back ends, as long as the back end can authenticate Keystone
> > credentials.  The only two back end options now in Castellan are
> > Barbican and a limited mock key manager useful only for unit tests.
> > If someone wrote a Keystone auth plugin for Vault, we could also have a
> > Vault back end for Castellan.
> > 
> > The benefit of using Castellan versus using Barbican directly
> > is Castellan allows the option of swapping out for other key managers,
> > mainly for testing.  If projects want their own custom back end for
> > Castellan, they can write a back end that implements the Castellan
> > interface but lives in their own code base, i.e., ConfKeyManager in
> > Nova and Cinder. Additionally, Castellan already has oslo.config
> > options defined which are helpful for configuring the project to talk
> > to Barbican.
> > 
> > When the Barbican team first created the Castellan library, we had
> > reached out to oslo to see if we could name it oslo.keymanager, but the
> > idea was not accepted because the library didn't have enough traction.
> > Now, Castellan is used in many projects, and we thought we would
> > suggest renaming again.  At the PTG, the Barbican team met with the AWG
> > to discuss how we could get Barbican integrated with more projects, and
> > the rename was also suggested at that meeting.  Other projects are
> > interested in creating encryption features, and a rename will help
> > clarify the difference between Barbican and Castellan.
> 
> Can you expand on why you think that is so? I'm not disagreeing with the
> statement, but it's not obviously true to me, either. I vaguely remember
> having it explained at the PTG, but I don't remember the details.
> 

To me, Oslo is a bunch of libraries that encompass "the way OpenStack
does ". When  is key management, projects are, AFAICT, universally
using Castellan at the moment. So I think it fits in Oslo conceptually.

As far as what benefit there is to renaming it, the biggest one is
divesting Castellan of the controversy around Barbican. There's no
disagreement that explicitly handling key management is necessary. There
is, however, still hesitance to fully adopt Barbican in that role. In
fact I heard about some alternatives to Barbican, namely "Vault"[1] and
"Tang"[2], that may be useful for subsets of the community, or could
even grow into de facto standards for key management.

So, given that there may be other backends, and the developers would
like to embrace that, I see value in renaming. It would help, I think,
Castellan's developers to be able to focus on key management and not
have to explain to every potential user "no we're not Barbican's cousin,
we're just an abstraction..".

> > Existing similar libraries (if any) and why they aren't being used: N/A
> > 
> > Reviewer activity: Barbican team
> 
> If the review team is going to be largely the same, I'm not sure I
> see the benefit of changing the ownership of the library. We certainly
> have other examples of Oslo libraries being managed mainly by
> sub-teams made up of folks who primarily focus on other projects.
> oslo.policy and oslo.versionedobjects come to mind, but in both of
> those cases the code was incubated in Oslo or brought into Oslo
> before the tools for managing shared libraries were widely used
> outside of the Oslo team. We now have quite a few examples of project
> teams managing shared libraries (other than their clients).
> 

While this makes sense, I'm not so sure any of those are actually
specifically in the same category as Castellan. Perhaps you can expand
on which libraries have done this, and how they're similar to Castellan?

[1] https://www.vaultproject.io/
[2] https://github.com/latchset/tang

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Small steps for Go

[Clint Byrum]

Excerpts from Davanum Srinivas's message of 2017-03-13 14:32:16 -0400:
> Clint,
> 
> There's some discussion on the etherpad, don't want to move that here.
> 

Ok. I don't see much that explains why. But either way, IMO this is
why an etherpad is a really bad place to have a discussion. Great for
recording notes _during_ a discussion. But it's not like I have a log
of what was said when by who.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

I know it happened a few messages up in the thread, but you may have
forgotten where I said that we should resurrect the idea of having
automatic instance-users as a feature of Nova (and even to add this to
Shade/Oaktree, so people could add it to existing OpenStack clouds
before they roll out the version that has it supported natively.

My only point there was that currently there is a workaround available,
and it wouldn't be that alien to most users of clouds.

Excerpts from Fox, Kevin M's message of 2017-03-13 14:31:10 +:
> So for cloud apps, you must have a config management system to do secure 
> secret transport? A cloud app developer must target which config management 
> tool? How do you know what the end user sites are running? Do you have to 
> support all of them? Config management tooling is very religious, so if you 
> don't pick the right one, folks will shun your app. Thats way too burdensome 
> on app developers and users to require.
> 
> Instead, look at how aws does creds, and k8s. Any vm can just request a fresh 
> token. In k8s, a fresh token is injected into the container automatically. 
> This makes it extremely easy to deal with. This is what app developers are 
> expecting now and openstack is turning folks away when we keep pushing a lot 
> of work their way.
> 
> Openstacks general solution for any hard problem seems to be, make the 
> operator, app dev, or user deal with it, not openstack.
> 
> Thats ok if those folks have no where else to go. They do now, and are 
> starting to do so as there are more options.
> 
> Openstack needs to abandon that phylosophy. It can no longer afford it.
> 
> Thanks,
> Kevin
> 
> 
> From: Clint Byrum
> Sent: Sunday, March 12, 2017 10:30:49 AM
> To: openstack-dev
> Subject: Re: [openstack-dev] [tc][appcat] The future of the App Catalog
> 
> Excerpts from Fox, Kevin M's message of 2017-03-12 16:54:20 +:
> > I totally agree that policy management is a major problem too. A much 
> > bigger one then instance users, and something I was hoping to get to after 
> > instance users, but never made it past the easier of the two. :/
> >
> >
> > The just inject creds solution keeps getting proposed, but only looks at 
> > the surface of the issue so has a lot of issues under the hood. Lets dig in 
> > again.
> >
> > Lets say I create a heat template and inject creds through Parameters, as 
> > that is the natural place for a user to fill out settings and launch their 
> > application.
> >
> > The cred is loaded unencrypted into the heat database. Then heat-engine 
> > pushes it into the nova database where it resides unencrypted, so it can be 
> > sent to cloud init, usually also in an unencrypted form.
> >
> > You delete the heat stack, and the credential still sticks around in the 
> > nova database long after the vm is deleted, as it keeps deleted data.
> >
> > The channels for passing stuff to a vm are much better at passing config to 
> > the vm, not secrets.
> >
> > Its also a one shot way to get an initial cred to a vm, but not a way to 
> > update it should the need arise. Also, how is the secret maintained in a 
> > way that rebooting the vm works while snapshotting the vm doesn't capture 
> > the secret, etc.
> >
> > The use case/issues are described exhaustively in the spec and describe why 
> > its not something thats something that can easily be tackled by "just do X" 
> > solutions. I proposed one implementation I think will work generally and 
> > cover all bases. But am open to other implementations that cover all the 
> > bases. Many half solutions have been proposed, but the whole point is 
> > security, so a half solution that has big security holes in it isn't really 
> > a solution.
> >
> 
> _OR_, you inject a nonce that is used to authenticate the instance to
> config management. If you're ever going to talk to anything outside of
> the cloud APIs, you'll need this anyway.
> 
> Once you're involved with config management you are already sending
> credentials of various types to your instances.
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Small steps for Go

[Clint Byrum]

Excerpts from Davanum Srinivas's message of 2017-03-13 10:06:30 -0400:
> Update:
> 
> * We have a new git repo (EMPTY!) for the commons work -
> http://git.openstack.org/cgit/openstack/golang-commons/
> * The golang-client has little code, but lot of potential -
> https://git.openstack.org/cgit/openstack/golang-client/
> 

So, we're going to pretend gophercloud doesn't exist and continue to
isolate ourselves from every other community?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Fox, Kevin M's message of 2017-03-12 16:54:20 +:
> I totally agree that policy management is a major problem too. A much bigger 
> one then instance users, and something I was hoping to get to after instance 
> users, but never made it past the easier of the two. :/
> 
> 
> The just inject creds solution keeps getting proposed, but only looks at the 
> surface of the issue so has a lot of issues under the hood. Lets dig in again.
> 
> Lets say I create a heat template and inject creds through Parameters, as 
> that is the natural place for a user to fill out settings and launch their 
> application.
> 
> The cred is loaded unencrypted into the heat database. Then heat-engine 
> pushes it into the nova database where it resides unencrypted, so it can be 
> sent to cloud init, usually also in an unencrypted form.
> 
> You delete the heat stack, and the credential still sticks around in the nova 
> database long after the vm is deleted, as it keeps deleted data.
> 
> The channels for passing stuff to a vm are much better at passing config to 
> the vm, not secrets.
> 
> Its also a one shot way to get an initial cred to a vm, but not a way to 
> update it should the need arise. Also, how is the secret maintained in a way 
> that rebooting the vm works while snapshotting the vm doesn't capture the 
> secret, etc.
> 
> The use case/issues are described exhaustively in the spec and describe why 
> its not something thats something that can easily be tackled by "just do X" 
> solutions. I proposed one implementation I think will work generally and 
> cover all bases. But am open to other implementations that cover all the 
> bases. Many half solutions have been proposed, but the whole point is 
> security, so a half solution that has big security holes in it isn't really a 
> solution.
> 

_OR_, you inject a nonce that is used to authenticate the instance to
config management. If you're ever going to talk to anything outside of
the cloud APIs, you'll need this anyway.

Once you're involved with config management you are already sending
credentials of various types to your instances.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Fox, Kevin M's message of 2017-03-11 21:31:40 +:
> No, they are treated as second class citizens. Take Trova again as an 
> example. The underlying OpenStack infrastructure does not provide a good 
> security solution for Trove's use case. As its more then just IaaS. So they 
> have spent years trying to work around it on one way or another, each with 
> horrible trade offs.
> 
> For example they could fix an issue by:
> 1. Run the service vm in the users tenant where it belongs. Though, currently 
> the user has permissions to reboot the vm, login through the console and 
> swipe any secrets that are on the vm and make it much harder for the cloud 
> admin to administer.
> 2. Run the vm in a "trove" tenant. This fixes the security issue but breaks 
> the quota model of OpenStack. Users with special host aggregate 
> access/flavors can't work with this model.
> 
> For our site, we can't use Trove at all at the moment, even though we want 
> to. Because option 2 doesn't work for us, and option 1 currently has a 
> glaring security flaw in it.
> 
> One of the ways I saw Trove try to fix it was to get a feature into Nova 
> called "Service VM's". VMs owned by the user but not fully controllable by 
> them but from some other OpenStack service on their behalf. This, IMO is the 
> right way to solve it. There are a lot of advanced services that need this 
> functionality. But it seems to have been rejected, as "users don't need 
> that"... Which is true, only if you only consider the IaaS use case.
> 

You're right. This type of rejection is not O-K IMO, because this is
consumers of Nova with a real use case, asking for real features that
simply cannot be implemented anywhere except inside Nova. Perhaps the
climate has changed, and this effort can be resurrected.

> The problems of these other OpenStack services are being rejected as second 
> class problems, not primary ones.
> 
> I'm sure other sites are avoiding other OpenStack advanced services for 
> similar reasons. its not just that Operators don't want to deploy it, or that 
> Users are not asking for it.
> 
> Let me try and explain Zane's post in a sligtly different way... maybe that 
> would help...
> 
> So, say you had an operating system. It had the ability to run arbitrary 
> programs if the user started an executable via the keyboard/mouse. But had no 
> ability for an executable to start another executable. How useful would that 
> OS be? There would be no shell scripts. No non monolithic applications. It 
> would be sort of functional, but would be hamstrung.
> 
> OpenStack is like that today. Like the DOS operating system. Programs are 
> expected to be pretty self contained and not really talk back to the 
> Operating System its running on, nor a way to discover other programs running 
> on the same system. Nor really for a script running on the Operating System 
> to start other programs, chaining them together in a way thats more useful 
> then the sum of their parts. The current view is fine if you need is just a 
> container to run a traditional OS in. Its not if you are trying to build an 
> application that spans things.
> 
> There have been several attempts at fixing this, in Heat, in Murano, in the 
> App Catalog, but plumbing they rely on isn't really supportive of it, as they 
> believe the use case really is just launching a VM with an OS in it is really 
> the important thing to do, and the job's done.
> 
> For the Applications Catalog to be successful, it needs the underlying cloud 
> to have enough functionality among a common set of cloud provided services to 
> allow applications developers to write cloud software that is redistributable 
> and consumable by the end user. Its failed because the infrastructure just 
> isn't there. The other advanced services are suffering from it too.
> 

I'm not sure I agree. One can very simply inject needed credentials
into a running VM and have it interact with the cloud APIs. However,
there is a deficiency that affects all VM users that might make this
a more desirable option.

There's a lack of a detailed policy management piece. What I'd really
like to inject is an API key that can only do the 2 things my app needs
(like, scale up load balancer, or allocate and attach a volume to itself).
Roles are just too opaque for that to really work well these days.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Christopher Aedo's message of 2017-03-10 19:30:18 -0800:
> On Fri, Mar 10, 2017 at 6:20 PM, Clint Byrum  wrote:
> > Excerpts from Fox, Kevin M's message of 2017-03-10 23:45:06 +:
> >> So, this is the kind of thinking I'm talking about... OpenStack today is
> >> more then just IaaS in the tent. Trove (DBaaS), Sahara (Hadoop,Spark,etc
> >> aaS), Zaqar (Messaging aaS) and many more services. But they seem to be
> >> treated as second class citizens, as they are not "IaaS".
> >>
> >
> > It's not that they're second class citizens. It's that their community
> > is smaller by count of users, operators, and developers. This should not
> > come as a surprise, because the lowest common denominator in any user
> > base will always receive more attention.
> >
> >> > Why should it strive to be anything except an excellent building block
> >> for other technologies?
> >>
> >> You misinterpret my statement. I'm in full agreement with you. The
> >> above services should be excellent building blocks too, but are suffering
> >> from lack of support from the IaaS layer. They deserve the ability to
> >> be excellent too, but need support/vision from the greater community
> >> that hasn't been forthcoming.
> >>
> >
> > You say it like there's some over arching plan to suppress parts of the
> > community and there's a pack of disgruntled developers who just can't
> > seem to get OpenStack to work for Trove/Sahara/AppCatalog/etc.
> >
> > We all have different reasons for contributing in the way we do.  Clearly,
> > not as many people contribute to the Trove story as do the pure VM-on-nova
> > story.
> >
> >> I agree with you, we should embrace the container folks and not treat
> >> them as separate. I think thats critical if we want to allow things
> >> like Sahara or Trove to really fulfil their potential. This is the path
> >> towards being an OpenSource AWS competitor, not just for being able to
> >> request vm's in a cloudy way.
> >>
> >> I think that looks something like:
> >> OpenStack Advanced Service (trove, sahara, etc) -> Kubernetes ->
> >> Nova VM or Ironic Bare Metal.
> >>
> >
> > That's a great idea. However, AFAICT, Nova is _not_ standing in Trove,
> > Sahara, or anyone else's way from doing this. Seriously, try it. I'm sure
> > it will work.  And in so doing, you will undoubtedly run into friction
> > from the APIs. But unless you can describe that _now_, you have to go try
> > it and tell us what broke first. And then you can likely submit feature
> > work to nova/neutron/cinder to make it better. I don't see anything in
> > the current trajectory of OpenStack that makes this hard. Why not just do
> > it? The way you ask, it's like you have a team of developers just sitting
> > around shaving yaks waiting for an important OpenStack development task.
> >
> > The real question is why aren't Murano, Trove and Sahara in most current
> > deployments? My guess is that it's because most of our current users
> > don't feel they need it. Until they do, Trove and Sahara will not be
> > priorities. If you want them to be priorities _pay somebody to make them
> > a priority_.
> 
> This particular point really caught my attention.  You imply that
> these additional services are not widely deployed because _users_
> don't want them.  The fact is most users are completely unaware of
> them because these services require the operator of the cloud to
> support them.  In fact they often require the operator of the cloud to
> support them from the initial deployment, as these services (and
> *most* OpenStack services) are frighteningly difficult to add to an
> already deployed cloud without downtime and high risk of associated
> issues.
> 
> I think it's unfair to claim these services are unpopular because
> users aren't asking for them when it's likely users aren't even aware
> of them (do OVH, Vexxhost, Dreamhost, Raskspace or others provide a
> user-facing list of potential OpenStack services with a voting option?
>  Not that I've ever seen!)
> 
> I bring this up to point out how much more popular ALL of these
> services would be if the _users_ were able to enable them without
> requiring operator intervention and support.
> 
> Based on our current architecture, it's nearly impossible for a new
> project to be deployed on a cloud without cloud-level admin
> privileges.  Additionall

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Zane Bitter's message of 2017-03-10 19:09:42 -0500:
> On 10/03/17 12:27, Monty Taylor wrote:
> > On 03/10/2017 10:59 AM, Clint Byrum wrote:
> You may be familiar with the Kuryr project, which integrates Kubernetes 
> deployments made by Magnum with Neutron networking so that other Nova 
> servers can talk directly to the containers and other fun stuff. IMHO 
> it's exactly the kind of thing OpenStack should be doing to make users' 
> lives better, and give a compelling reason to install k8s on top of 
> OpenStack instead of on bare metal.
> 
> So here's a fun thing I learned at the PTG: according to the Magnum 
> folks, the main thing preventing them from fully adopting Kuryr is that 
> the k8s application servers, provisioned with Nova, need to make API 
> calls to Neutron to set up the ports as containers move around. And 
> there's no secure way to give Keystone authentication credentials to an 
> application server to do what it needs - and, especially, to do *only* 
> what it needs.
> 
> http://lists.openstack.org/pipermail/openstack-dev/2016-October/105304.html
> 
> Keystone devs did agree in back in Austin that when they rejigger the 
> default policy files it will done in such a way as to make the 
> authorisation component of this feasible (by requiring a specific 
> reader/writer role, not just membership of the project, to access APIs), 
> but that change hasn't happened yet AFAIK. I suspect that it isn't their 
> top priority. Kevin has been campaigning for *years* to get Nova to 
> provide a secure way to inject credentials into a server in the same way 
> that this is built in to EC2, GCE and (I assume but haven't checked) 
> Azure. And they turned him down flat every time saying that this was not 
> Nova's problem.
> 
> Sorry, but if OpenStack isn't a good, secure platform for running 
> Kubernetes on then that is a HAIR ON FIRE-level *existential* problem in 
> 2017.
> 

You're very right on this. And kuryr not working the way it should is _a
disaster_. I was hoping it had worked itself out by now.

I've argued against certain aspects of instance-users in the past (and
for others). I think it might be worth it to take another shot at
actually writing up a spec again.

In the mean time, we could always have shade inject instance-user
creds... 

> We can't place too much blame on individual projects though, because I 
> believe the main reason this doesn't Just Work already is that there has 
> been an unspoken consensus that we needed to listen to users like you 
> but not to users like Kevin, and the elected leaders of our community 
> have done nothing to either disavow or officially adopt that consensus. 
> We _urgently_ need to decide if that's what we actually want and make 
> sure it is prominently documented so that both users and developers know 
> what's what.
> 
> FWIW I'm certain you must have hit this same issue in infra - probably 
> you were able to use pre-signed Swift URLs when uploading log files to 
> avoid needing credentials on servers allocated by nodepool? That's 
> great, but not every API in OpenStack has a pre-signed URL facility, and 
> nor should they have to.
> 

Indeed, that's how infra nodes store logs in swift.

> (BTW I proposed a workaround for Magnum/Kuryr at the PTG by using a 
> pre-signed Zaqar URL with a subscription triggering a Mistral workflow, 
> and I've started working on a POC.)
> 

What triggers the boot to kick over the bucket full of golf balls
though?

> > Considering computers as some-how inherently 'better' or 'worse' than
> > some of the 'cloud-native' concepts is hog wash. Different users have
> > different needs. As Clint points out - kubernetes needs to run
> > _somewhere_. CloudFoundry needs to run _somewhere_. So those are at
> > least two other potential users who are not me and my collection of
> > things I want to run that want to run in computers.
> 
> I think we might be starting to talk about different ideas. The whole 
> VMs vs. containers fight _is_ hogwash. You're right to call it out as 
> such. We hear far too much about it, and it's totally unfair to the 
> folks who work on the VM side. But that isn't what this discussion is about.
> 
> Google has done everyone a minor disservice by appropriating the term 
> "cloud-native" and using it in a context such that it's effectively been 
> redefined to mean "containers instead of VMs". I've personally stopped 
> using the term because it's more likely to generate confusion than clarity.
> 
> What "cloud-native" used to mean to me was an application 

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Fox, Kevin M's message of 2017-03-10 23:45:06 +:
> So, this is the kind of thinking I'm talking about... OpenStack today is
> more then just IaaS in the tent. Trove (DBaaS), Sahara (Hadoop,Spark,etc
> aaS), Zaqar (Messaging aaS) and many more services. But they seem to be
> treated as second class citizens, as they are not "IaaS".
> 

It's not that they're second class citizens. It's that their community
is smaller by count of users, operators, and developers. This should not
come as a surprise, because the lowest common denominator in any user
base will always receive more attention.

> > Why should it strive to be anything except an excellent building block
> for other technologies?
> 
> You misinterpret my statement. I'm in full agreement with you. The
> above services should be excellent building blocks too, but are suffering
> from lack of support from the IaaS layer. They deserve the ability to
> be excellent too, but need support/vision from the greater community
> that hasn't been forthcoming.
> 

You say it like there's some over arching plan to suppress parts of the
community and there's a pack of disgruntled developers who just can't
seem to get OpenStack to work for Trove/Sahara/AppCatalog/etc.

We all have different reasons for contributing in the way we do.  Clearly,
not as many people contribute to the Trove story as do the pure VM-on-nova
story.

> I agree with you, we should embrace the container folks and not treat
> them as separate. I think thats critical if we want to allow things
> like Sahara or Trove to really fulfil their potential. This is the path
> towards being an OpenSource AWS competitor, not just for being able to
> request vm's in a cloudy way.
> 
> I think that looks something like:
> OpenStack Advanced Service (trove, sahara, etc) -> Kubernetes ->
> Nova VM or Ironic Bare Metal.
> 

That's a great idea. However, AFAICT, Nova is _not_ standing in Trove,
Sahara, or anyone else's way from doing this. Seriously, try it. I'm sure
it will work.  And in so doing, you will undoubtedly run into friction
from the APIs. But unless you can describe that _now_, you have to go try
it and tell us what broke first. And then you can likely submit feature
work to nova/neutron/cinder to make it better. I don't see anything in
the current trajectory of OpenStack that makes this hard. Why not just do
it? The way you ask, it's like you have a team of developers just sitting
around shaving yaks waiting for an important OpenStack development task.

The real question is why aren't Murano, Trove and Sahara in most current
deployments? My guess is that it's because most of our current users
don't feel they need it. Until they do, Trove and Sahara will not be
priorities. If you want them to be priorities _pay somebody to make them
a priority_.

> Not what we have today:
> OpenStack Advanced Service -> Nova VM or Ironic Bare Metal
> 
> due to the focus on the api's of VM's being only for IaaS and not for
> actually running cloud software on.
> 

The above is an unfounded and unsupported claim. What _exactly_ would
you want Nova/Neutron/Cinder's API to do differently to support "cloud
software" better? Why is everyone dodging that question? Name one
specific change and how it would actually be consumed.

I personally think it's just the general frustration that comes at
this stage of maturity of a project. There's a contraction of hype,
lower investment in general, and everyone is suppressing their panic
and trying to reason about what we can do to make it better, but nobody
actually knows. Meanwhile, our users and operators need us to keep
making things better. Some are even _paying us_ to make things better.

> I'm sorry if that sounds a bit confusing. Its hard to
> explain. I can try and elaborate if you want. Zane's
> posting here does help explain it a little: >
> http://www.zerobanana.com/archive/2015/04/24#a-vision-for-openstack
> 

Honest, I don't understand the reality that Zane's vision is driving at in
that post. More Zaqar? Why not just do that? What's standing in the way?

> The other alternative is to clearly define OpenStack to be just IaaS,
> and kick all the non IaaS stuff out of the tent. (I do not prefer
> this). It will hurt in the short term but long tern could be better
> for those projects then the current status quo as new opportunities for
> switching base dependencies will present themselves and no longer will
> be hamstrung by those that feel their use case isn't important or think
> that the existing api's are "good enough" to handle the use case.
> 

Why would we kick out perfectly healthy pieces of software that want
to be OpenStack-native? Nobody is saying that IaaS clouds aren't well
complimented by native higher level projects. But in the app catalog
case, there's little consumption, and waning contribution, so it's worth
considering whether its continued maintenance and existence is a drain
on the overall community. Sounds like it is, and we'll proba

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Fox, Kevin M's message of 2017-03-10 20:47:55 +:
> Just because the market hasn't immediately shifted from IaaS to containers 
> doesn't mean it won't happen eventually, and that google's wrong in their 
> push for containers over IaaS. It took a long time (still ongoing) to move 
> from physical machines to vm's. The same parallel will happen with containers 
> I believe. We're at least a few years out from it becoming more common place. 
> That doesn't mean folks can assume it will always be the way it is, and the 
> "market has spoken". "The only constants are death and taxes." :)
> 
> Your right in the assertion k8s needs a place to run, but that doesn't 
> necessarily mean OpenStack, unless we help integrate it and make it a great 
> place to run it.
> 
> I'm glad the IaaS stuff in OpenStack suits your needs. Thats great. It hasn't 
> served a great number of users needs though, and has been at least misleading 
> folks into believing it will for a number of years. If we want it to just be 
> an IaaS, lets free up those users to move on elsewhere.
> 
> Does OpenStack intend just to be an IaaS in a few years or something else?
> 

Why should it strive to be anything except an excellent building block
for other technologies?

Containers have a community and there's no reason we should think of that
as separate from us. We're friends, and we overlap when it makes sense.

All the container tech that I see out there still needs computers /
disks / networks to run on. OpenStack is a pretty decent way to chop your
computers up and give fully automated control of all aspects of them to
multiple tenants.  What else would you want it to do that isn't already
an aspect of Kubernetes, CloudFoundry, etc.?

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Joshua Harlow's message of 2017-03-10 10:09:24 -0800:
> Clint Byrum wrote:
> > Excerpts from Joshua Harlow's message of 2017-03-09 21:53:58 -0800:
> >> Renat Akhmerov wrote:
> >>>> On 10 Mar 2017, at 06:02, Zane Bitter >>>> <mailto:zbit...@redhat.com>>  wrote:
> >>>>
> >>>> On 08/03/17 11:23, David Moreau Simard wrote:
> >>>>> The App Catalog, to me, sounds sort of like a weird message that
> >>>>> OpenStack somehow requires applications to be
> >>>>> packaged/installed/deployed differently.
> >>>>> If anything, perhaps we should spend more effort on advertising that
> >>>>> OpenStack provides bare metal or virtual compute resources and that
> >>>>> apps will work just like any other places.
> >>>> Look, it's true that legacy apps from the 90s will run on any VM you
> >>>> can give them. But the rest of the world has spent the last 15 years
> >>>> moving on from that. Applications of the future, and increasingly the
> >>>> present, span multiple VMs/containers, make use of services provided
> >>>> by the cloud, and interact with their own infrastructure. And users
> >>>> absolutely will need ways of packaging and deploying them that work
> >>>> with the underlying infrastructure. Even those apps from the 90s
> >>>> should be taking advantage of things like e.g. Neutron security
> >>>> groups, configuration of which is and will always be out of scope for
> >>>> Docker Hub images.
> >>>>
> >>>> So no, we should NOT spend more effort on advertising that we aim to
> >>>> become to cloud what Subversion is to version control. We've done far
> >>>> too much of that already IMHO.
> >>> 100% agree with that.
> >>>
> >>> And this whole discussion is taking me to the question: is there really
> >>> any officially accepted strategy for OpenStack for 1, 3, 5 years?
> >> I can propose what I would like for a strategy (it's not more VMs and
> >> more neutron security groups...), though if it involves (more) design by
> >> committee, count me out.
> >>
> >> I honestly believe we have to do the equivalent of a technology leapfrog
> >> if we actually want to be relevant; but maybe I'm to eager...
> >>
> >
> > Open source isn't really famous for technology leapfrogging.
> 
> Time to get famous.
> 
> I hate accepting what the status quo is just because it's not been 
> famous (or easy, or turned out, or ...) before.
> 

Good luck. I can't see how you get an investor to enable you to do that
in this context without an absolute _mountain_ of relatively predictable
service-industry profit involved.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tc][appcat][murano][app-catalog] The future of the App Catalog

[Clint Byrum]

Excerpts from Thierry Carrez's message of 2017-03-10 16:48:02 +0100:
> Christopher Aedo wrote:
> > On Thu, Mar 9, 2017 at 4:08 AM, Thierry Carrez  
> > wrote:
> >> Christopher Aedo wrote:
> >>> On Mon, Mar 6, 2017 at 3:26 AM, Thierry Carrez  
> >>> wrote:
>  [...]
>  In parallel, Docker developed a pretty successful containerized
>  application marketplace (the Docker Hub), with hundreds of thousands of
>  regularly-updated apps. Keeping the App Catalog around (including its
>  thinly-wrapped Docker container Murano packages) make us look like we
>  are unsuccessfully trying to compete with that ecosystem, while
>  OpenStack is in fact completely complementary.
> >>>
> >>> Without something like Murano "thinly wrapping" docker apps, how would
> >>> you propose current users of OpenStack clouds deploy docker apps?  Or
> >>> any other app for that matter?  It seems a little unfair to talk about
> >>> murano apps this way when no reasonable alternative exists for easily
> >>> deploying docker apps.  When I look back at the recent history of how
> >>> we've handled containers (nova-docker, magnum, kubernetes, etc) it
> >>> does not seem like we're making it easy for the folks who want to
> >>> deploy a container on their cloud...
> >>
> >> I'd say there are two approaches: you can use the container-native
> >> approach ("docker run" after provisioning some container-enabled host
> >> using Nova or K8s cluster using Magnum), or you can use the
> >> OpenStack-native approach (zun create nginx) and have it
> >> auto-provisioned for you. Those projects have a narrower scope, and
> >> fully co-opt the container ecosystem without making us appear as trying
> >> to build our own competitive application packaging/delivery/marketplace
> >> mechanism.
> >>
> >> I just think that adding the Murano abstraction in the middle of it and
> >> using an AppCatalog-provided Murano-powered generic Docker container
> >> wrapper is introducing unnecessary options and complexity -- options
> >> that are strategically hurting us when we talk to those adjacent
> >> communities...
> > 
> > OK thank you for making it clearer, now I understand where you're
> > coming from.  I do agree with this sentiment.  I don't have any
> > experience with zun but it sounds like it's the least-cost way to
> > deploy a docker at for the environments where it's installed.
> > 
> > I think overall the app catalog was an interesting experiment, but I
> > don't think it makes sense to continue as-is.  Unless someone comes up
> > with a compelling new direction, I don't see much point in keeping it
> > running.  Especially since it sounds like Mirantis is on board (and
> > the connection to a murano ecosystem was the only thing I saw that
> > might be interesting).
> 
> Right -- it's also worth noting that I'm only talking about the App
> Catalog here, not about Murano. Zun might be a bit too young for us to
> place all our eggs in the same basket, and some others pointed to how
> Murano is still viable alternative package for things that are more
> complex than a set of containers. What I'm questioning at the moment is
> the continued existence of a marketplace that did not catch fire as much
> as we hoped -- an app marketplace with not enough apps just hurts more
> than it helps imho.
> 
> In particular, I'm fine if (for example) the Docker-wrapper murano
> package ends up being shipped as a standard/example package /in/ Murano,
> and continues to exist there as a "reasonable alternative for easily
> deploying docker apps" :)
> 

While we were debating how to do everything inside our walls, Google
and Microsoft became viable public cloud vendors along side the other
players. We now live in a true multi-cloud world (not just a theoretical
one).

And what I see when I look outside our walls is not people trying to make
the initial steps identical or easy. For that there's PaaS. Instead, for
those that want the full control of their computers that IaaS brings,
there's a focus on making it simple, and growing a process that works
the same for the parts that are the same, and differently for the parts
that are different.

I see things like Terraform embracing the differences in clouds, not
hiding behind lowest common denominators. So if you want a Kubernetes on
GCE and one on OpenStack, you'd write two different Terraform plans
that give you the common set of servers you expect, get you config
management and kubernetes setup and hooked into the infrastructure
however it needs to be, and then get out of your way.

So, while I think it's cool to make sure we are supporting our users
when all they want is us, it might make more sense to do that outside
our walls, where we can meet the rest of the cloud world too.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.o

Re: [openstack-dev] [tc][appcat] The future of the App Catalog

[Clint Byrum]

Excerpts from Joshua Harlow's message of 2017-03-09 21:53:58 -0800:
> Renat Akhmerov wrote:
> >
> >> On 10 Mar 2017, at 06:02, Zane Bitter  >> > wrote:
> >>
> >> On 08/03/17 11:23, David Moreau Simard wrote:
> >>> The App Catalog, to me, sounds sort of like a weird message that
> >>> OpenStack somehow requires applications to be
> >>> packaged/installed/deployed differently.
> >>> If anything, perhaps we should spend more effort on advertising that
> >>> OpenStack provides bare metal or virtual compute resources and that
> >>> apps will work just like any other places.
> >>
> >> Look, it's true that legacy apps from the 90s will run on any VM you
> >> can give them. But the rest of the world has spent the last 15 years
> >> moving on from that. Applications of the future, and increasingly the
> >> present, span multiple VMs/containers, make use of services provided
> >> by the cloud, and interact with their own infrastructure. And users
> >> absolutely will need ways of packaging and deploying them that work
> >> with the underlying infrastructure. Even those apps from the 90s
> >> should be taking advantage of things like e.g. Neutron security
> >> groups, configuration of which is and will always be out of scope for
> >> Docker Hub images.
> >>
> >> So no, we should NOT spend more effort on advertising that we aim to
> >> become to cloud what Subversion is to version control. We've done far
> >> too much of that already IMHO.
> >
> > 100% agree with that.
> >
> > And this whole discussion is taking me to the question: is there really
> > any officially accepted strategy for OpenStack for 1, 3, 5 years?
> 
> I can propose what I would like for a strategy (it's not more VMs and 
> more neutron security groups...), though if it involves (more) design by 
> committee, count me out.
> 
> I honestly believe we have to do the equivalent of a technology leapfrog 
> if we actually want to be relevant; but maybe I'm to eager...
> 

Open source isn't really famous for technology leapfrogging.

And before you say "but Docker.." remember that Solaris had zones 13
years ago.

What a community like ours is good at doing is gathering all the
exciting industry leading bleeding edge chaos into a boring commodity
platform. What Zane is saying (and I agree with) is let's make sure we see
the whole cloud forest and not just focus on the VM trees in front of us.

I'm curious what you (Josh) or Zane would change too.
Containers/apps/kubes/etc. have to run on computers with storage and
networks. OpenStack provides a pretty rich set of features for giving
users computers with storage on networks, and operators a way to manage
those. So I fail to see how that is svn to "cloud native"'s git. It
seems more foundational and complimentary.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Some information about the Forum at the Summit in Boston

[Clint Byrum]

Excerpts from Thierry Carrez's message of 2017-03-10 17:04:54 +0100:
> Ben Swartzlander wrote:
> > On 03/09/2017 12:10 PM, Jonathan Bryce wrote:
> >> Putting that aside, I appreciate your providing your input. The most
> >> consistent piece of feedback we received was around scheduling and
> >> visibility for sessions, so I think that is definitely an area for
> >> improvement at the next PTG. I heard mixed feedback on whether the
> >> ability to participate in multiple projects was better or worse than
> >> under the previous model, but understanding common conflicts ahead of
> >> time might give us a chance to schedule in a way that makes the
> >> multi-project work more possible. Did you participate in both Cinder
> >> and Manila mid-cycles in addition to the Design Summit sessions
> >> previously? Trying to understand which types of specific interactions
> >> you’re now less able to participate in.
> > 
> > Yes in the past I was able to attend all of the Manila and most of the
> > Cinder sessions at the Design summit, and I was able to attend the
> > Cinder midcycles in person and (since I'm the PTL) I was able to
> > schedule the Manila midcycles to not conflict.
> 
> On that particular bit of feedback ("making it impossible to participate
> in 2 or more vertical projects") it is feedback that I definitely heard :)
> 
> While the event structure made it generally a lot easier to tap into
> other teams (and a *lot* of them did just that), the horizontal/vertical
> split definitely made it more difficult for Manila folks to attend all
> Cinder sessions, or for Storlets folks to attend all Swift sessions. On
> a personal note, it made it more difficult for *me* to attend all
> Architecture WG and Stewardship workgroup and Release Team and Infra
> sessions, which were all going on at the same time on Monday/Tuesday. So
> it's not something that only affected vertical projects.
> 
> We can definitely improve on that, and come up with a more... creative
> way to split usage of rooms than a pretty-arbitrary grouping of projects
> into "vertical" and "horizontal" groups. There is no miracle solution
> (there will always be someone needing to be in two places at the same
> time), but the strawman split we tried for the first one is certainly
> not the optimal solution. If you have suggestions on how we can better
> map room/days, I'm all ears. I was thinking about taking input on major
> team overlaps (like the one you pointed to between Manila and Cinder) as
> a first step, and try to come up with a magic formula that would
> minimize conflicts.
> 

For those who didn't ever use it, attendees to UDS (Ubuntu Dev Summit)
would mark themselves as interested or required for sessions (in the
Launchpad blueprint system), and would express which days they'd be
at the summit.  Then a scheduling program would automatically generate
schedules with the least amount of conflicts.

I'm not saying we should copy summit's model, or (noo) use the
actual Summit django app[1]. But we _could_ use a similar algorithm,
except have project teams, instead of individuals, as the attendees,
perhaps splitting liasons/tc members/etc. off as their own groups,
and then at least have an optimal schedule generated.

A few summary points about UDS for those interested (tl;dr - It's not perfect)

 * UDS also wanted everyone to move around in the hallways every hour or
   so. There was a desire to _not_ let people "camp out" in one room. As
   a person who thrives in the bustling hallways, I like that. But those
   who need a quieter room to build confidence, and a more parliamentary
   procedure to get their voice heard are penalized. Also PTG is for
   focused hacking time too, but that could be solved by having large
   blocks for focus/pairing/etc time.

 * Running the summit scheduler in real-time as sessions and attendees
   were added created _unbelievable chaos_. There was almost always
   somebody cursing at the summit schedule screens as their most important
   session was moved to a small room, double-booked, or moved to a day
   they weren't going to be there. In my 7 or so UDS's, I think we only
   tried that for 2 of them before it was locked down the week before UDS.

 * Not running the summit scheduler in real-time meant that added
   sessions and new attendees were at a disadvantage and had to manually
   try to coordinate with the free space on the schedule. Since the
   schedule was tuned to the more static attendee base and set of
   sessions, this usually meant that the hotel bar after sessions was
   the more reliable place to have a discussion that wasn't expected.

[1] https://launchpad.net/summit

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Small steps for Go

[Clint Byrum]

Excerpts from Michał Jastrzębski's message of 2017-03-09 11:23:17 -0800:
> Clint, I think it's good to find initial idea to work towards to
> create stuff like infrastructure (testing and such) around golang,
> code guidance, standards and so on. It takes first project to create
> all that stuff and it's good if it's bigger community effort. Next
> projects could just follow these standards really.
> 

I'm not sure I follow. There's already demand for golang infrastructure in
Designate and Swift, so I don't see why we wouldn't make sure our
infrastructure supports those.

> As for Kolla using golang - Kolla uses ansible and k8s today;) Or
> rather Kolla-ansible and Kolla-k8s uses Kolla. If you want to make
> Kolla-golang, you can...it just doesn't make any sense really (imho);)
> 

The etherpad made a veiled reference to Kolla wanting to do something in
Go. I was just making the point that I think deployment projects are
already other-language-based and so don't follow the same set of
constraints as the OpenStack project itself.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Some information about the Forum at the Summit in Boston

[Clint Byrum]

Excerpts from Ben Swartzlander's message of 2017-03-09 11:23:31 -0500:
> I might be the only one who has negative feelings about the PTG/Forum 
> split, but I suspect the foundation is suppressing negative feedback 
> from myself and other developers so I'll express my feelings here. If 
> there's anyone else who feels like me please reply, otherwise I'll 
> assume I'm just an outlier.
> 
> The new structure is asking developers to travel 4 times a year 
> (minimum) and makes it impossible to participate in 2 or more vertical 
> projects.
> 
> I know that most of the people working on Manila have pretty limited 
> travel budgets, and meeting 4 times a year basically guarantees that a 
> good number of people will be remote at any given meeting. From my 
> perspective if I'm going to be meeting with people on the phone I'd 
> rather be on the phone myself and have everyone on equal footing.
> 
> I also normally try to participate in Cinder as well as Manila and the 
> new PTG structures makes that impossible. I decided to try to be 
> positive and to wait until after the PTG to make up my mind but having 
> attended in Atlanta it was exactly as bad as I expected in terms of my 
> ability to participate in Cinder.
> 
> I will be in Boston to try to develop a firsthand opinion of the new 
> Forum format but as of now I'm pretty unhappy with the proposal. For 
> Manila I'm proposing that the community either meets at PTG and skips 
> conferences or meetings at conferences and skips PTGs going forward. I'm 
> not going to ask everyone to travel 4 times a year.
> 

If we reframe this as "how can I get the most out of this new situation"
instead of "why I don't like it" then I think you might find things to
like, and you might even find that despite the problems, it has a net
positive effect.

For instance, you are upset that you have to travel 4 times a year. But do
you need _every_ developer at the Forum to hear from operators and users?

Wouldn't you rather relieve those developers of that burden? A few of
you will have to go to the Forum, but in so doing, you should be able
to focus on the important aspect of it: USERS. If two Manila devs go to
the Forum, you can literally ignore each other the entire time and get
a massive amount of value. There's no more pressure pulling you apart in
10 directions like there was at the Design Summit.

Combine that with the lower cost and subsidized travel support program
for the PTG, and you should end up with a more complete gathering of
developers at PTG, and a better interface with users and operators. That
is the theory, and I think it's worth trying to fit one's approach into
that, rather than try to keep doing things the old way and expecting
this new system to work.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Small steps for Go

[Clint Byrum]

Excerpts from Hayes, Graham's message of 2017-03-07 14:19:02 +:
> On 07/03/2017 13:22, Davanum Srinivas wrote:
> > Folks,
> >
> > Anyone interested? https://etherpad.openstack.org/p/go-and-containers
> >
> > Thanks,
> > Dims
> >
> 
> The first thing that is required (according to the new guidelines) is
> for there to be a technical reason we need to use go, and cannot use
> python. [0]
> 
> This was the major point thrown at designate when we asked for
> permission - if there is now a golang project for the sake of
> a golang project, that would seem at odds with these (brand new)
> requirements.
> 
> The only item in the linked etherpad that would fit that requirement
> is the Go Common Lib, which is a requirement for any project who wants
> to use go.
> 
> Do we want to allow projects innovate, or do we just want Golang in
> OpenStack now? I honestly cannot follow what is going on in this area.
> 

There's a nasty assumption wrapped in that sentence that isn't quite
fair. We've always wanted projects to innovate, but the choice of common
language was entirely designed to put people in a box that was easier
for operators to consume and debug.

I don't see an actual real problem statement that takes operators and
users into account in that etherpad*. IMO language bindings is a red
herring, we have REST API's and they're not so bad that they have to
be wrapped in convenience libs *cough*OaktreeShade*cough*.

I'd really like to see *users* and/or *operators* be the focus of any
efforts around Golang, or Lisp, or anything else we ship.

Designate felt that the operators were better served by a Golang
transfer daemon (IIRC). Hummingbird was created by Swift operators to
deal with their IO concurrency problems. I _like_ those efforts (now
that I understand them, which was not easy). But the etherpad kind of
reads to me like "let's make Go things because container things".. maybe
somebody can explain it to me like I'm 5?


* Kolla is a deployment project, and imo, can do Golang the same way
  openstack-ansible does ansible and puppet does puppet.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][swg] per-project "Business only" moderated mailing lists

[Clint Byrum]

Excerpts from Jonathan Bryce's message of 2017-03-01 11:49:38 -0600:
> 
> > On Feb 28, 2017, at 4:25 AM, Thierry Carrez  wrote:
> > 
> > Clint Byrum wrote:
> >>>> So, I'll ask more generally: do you believe that the single openstack-dev
> >>>> mailing list is working fine and we should change nothing? If not, what
> >>>> problems has it created for you? 
> >>> 
> >>> As a person who sends a lot of process-driven email to this list,
> >>> it is not working for my needs to communicate with others.
> >>> 
> >>> Over the past few cycles when I was the release PTL, I always had
> >>> a couple of PTLs say there was too much email on this list for them
> >>> to read, and that they had not read my instructions for managing
> >>> releases. That resulted in us having to train folks at the last
> >>> minute, remind them of deadlines, deal with them missing deadlines,
> >>> and otherwise increased the release team's workload.
> >>> 
> >>> It is possible the situation will improve now that the automation
> >>> work is mostly complete and we expect to see fewer significant
> >>> changes in the release workflow. That still leaves quite a few
> >>> people regularly surprised by deadlines, though.
> >> 
> >> The problem above is really the krux of it. Whether or not you can keep
> >> up with the mailing list can be an unknown, unknown. Even now, those
> >> who can't actually handle the mailing list traffic are in fact likely
> >> missing this thread about whether or not people can handle the mailing
> >> list traffic (credit fungi for pointing out this irony to me on IRC).
> > 
> > Right, the main issue (for me) is that there is no unique way to reach
> > out to people that you're 100% sure they will read. For some the miracle
> > solution will be a personal email, for some it will be an IRC ping, for
> > some it will be a Twitter private message. There is no 100% sure
> > solution, and everyone prioritizes differently. The burden of reaching
> > out and making sure the message was acknowledged is on the person who
> > sends the message, and that just doesn't scale past 50 teams. That
> > includes release team communications to PTLs, but also things like
> > election nomination deadlines and plenty of other things.
> 
> Clint asked if there were specific issues in the workflow, and one item both 
> Thierry and Doug have identified is reaching ALL project leaders consistently 
> with important notifications or requests. I have also seen some working group 
> leaders and Foundation staff experience similar difficulties. Perhaps 
> creating a business-oriented list for PTLs similar to docs/infra that could 
> help with that particular problem.

Agreed. I think I may have even missed the krux of the reason for
the business lists, which was more "how do we get an important signal
through".

IMO this is where the announcement list would be useful. But that has
become something else entirely with release notifications (or it hasn't,
I don't know, I dropped it). But generally projects do have a low
traffic higher-priority list for announcements.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [barbican] Rolling upgrade in Barbican project

[Clint Byrum]

Excerpts from na...@vn.fujitsu.com's message of 2017-02-28 09:52:13 +:
> Hi everyone,
> 
> Recently, there are many emails to discuss a topic that "Why are projects 
> trying to avoid Barbican, still?" [0]. That is very an interesting topic. Now 
> I would like to make a new topic related to Rolling upgrade. I am trying to 
> find  information about the strategy to support rolling-upgrade for Barbican 
> project. Unfortunately, there is not any results, if any, could you please 
> update it for me. 
> 
> In my point of view, in order to support this feature, there will be three 
> impacts which need to solve:
> 
> 1. API versioning:
> 
> Maybe, Barbican will have version two [1] so I think we should have a plan to 
> still support version 1 on version 2. What do you about this point?
> 

Don't do it? Evolve v1 as needed, bud I'd recommend keeping whatever
users you currently have functional for as long as possible. Work with
the API-WG to make sure what you're doing is driving toward an API that
fits inside their best practices.

> 2. Database
> 
> - For OVO (Oslo version object) [2], I am wondering we should use OVO for 
> this project. In my option, I am afraid that OVO is overkill for this project.
> - For DB upgrading, currently there are some files [3-4] to drop and alter 
> column. This really should not have because there is not still have a 
> solution in case of delete or alter DB. That why there is a rule that don't 
> allow somebody to delete/alter DB in Nova and Neutron :) Do you think we 
> should prepare for this?
> 

We should write this down somewhere if we haven't already but these are
the basic principles that will let live upgrades happen in your
database:

* Add columns with default values or make them nullable
* Drop columns and tables only after all releases that reference them are EOL.
* Never rename anything. Create new, migrate data, drop old after EOL.
* Make new code able to detect a partial migration and fall back to old
  behavior.
  

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [architecture] Pike PTG Architecture Working Group Recap

[Clint Byrum]

The Architecture Working Group was lucky enough to have a room allocated
to us for all day Tuesday of the Pike PTG. We made good use of it
to do what we are here to do: Facilitate the development of a shared
architecture amongst OpenStack project teams.

We had an etherpad[1] that is a bit of a stream of consciousness, so
I'll attempt to distill it into topics:

Base Services
-

The base-services document recently landed in the governance repo[2],
defining the minimum things that any project can expect to have in a
deployment. The next step is to develop a process for evaluating the set
of base services. The topic of reducing "an oslo.db database" to just
"mysql" was mentioned, but that is happening more at the TC level and
was not something any of us in the arch-wg came prepared to discussion.


DLM
---

We did have a lively update on DLM adoption in OpenStack, and whether or
not it's time to have a base service discussion about DLM's (like
Zookeeper or etcd). There was some confusion as to whether Cinder needs
it. We learned that you can run Cinder without it, but having a DLM behind
Cinder allows you to run cinder-volume active/active/... vs. just
active/passive. Also drivers in Cinder can rely on a properly configured
DLM to protect remote resources, which is cool. This isn't really our
spec, but we want to make sure we help socialize it so that we can come
back and have the conversation about base services and DLMs later. There
was also some push back on that very idea from a few Nova developers who
felt that Nova does not need a DLM, but this isn't about projects who
don't need one, it's about those that were already identified long ago
in the spec that do need one.

Barbican


As part of the work to investigate new potential base services, we had a
lengthy discussion with the Barbican team and members of the security
team about whether projects need key management, and if they do, should
they use Barbican. This was complex, and probably needs a more detailed
write-up. Suffice to say, we do think that the Castellan library which
abstracts key management away from Barbican is a good start at having a
similar experience to "... an oslo.db database." As such, there will be
an effort made to look into renaming Castellan to "oslo.keymanager"
given that it is already in the dependency chain for several projects
and fits exactly into oslo's model of "the way OpenStack uses
keymanagers." There was some discussion about potentially using
non-Barbican key management solutions, such as just writing to
HashiCorp's Vault directly, or also looking at something like the
relatively untested, but very interesting "Tang/Clevis"[3] using the
McCallum-Relyea key exchange algorithm. It was a really interesting
conversation, and I expect more analysis to come out soon hopefully. For
now there aren't many actions to report.

Nova Compute API


I'll be honest, I don't really know what happened in that room. I was
hoping to get feedback from all of the projects, but it mostly turned
into Nova explaining to me all the abstractions they've added that make
nova-compute less of a layer violation than it used to be. I take the
relative silence of other projects as either Stockholm Syndrome, or
evidence that they agree and there's nothing to see here. In all, I
learned a lot and will produce some analysis myself soon based on the
facts we gathered in the etherpad[4] and in my brain.

Hierarchical Quotas
---

This wasn't really our show, but it happened in the Arch-WG space and I
learned a lot. I think Matt Riedemann already covered the prescient
points from this discussion. The etherpad [5] contains the details.

[1] https://etherpad.openstack.org/p/ptg-architecture-workgroup
[2] https://governance.openstack.org/tc/reference/base-services.html
[3] https://github.com/latchset/tang
[4] https://etherpad.openstack.org/p/arch-wg-nova-compute-api-ptg-pike
[5] https://etherpad.openstack.org/p/ptg-hierarchical-quotas

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][swg] per-project "Business only" moderated mailing lists

[Clint Byrum]

Excerpts from Doug Hellmann's message of 2017-02-27 15:43:12 -0500:
> Excerpts from Clint Byrum's message of 2017-02-27 09:35:21 -0800:
> > Excerpts from Dean Troyer's message of 2017-02-27 09:32:09 -0600:
> > > On Mon, Feb 27, 2017 at 3:31 AM, Clint Byrum  wrote:
> > > > This is not for users who only want to see some projects. That is a well
> > > > understood space and the mailman filtering does handle it. This is for
> > > > those who want to monitor the overall health of the community, address
> > > > issues with cross-project specs, or participate in so many projects it
> > > > makes little sense to spend time filtering.
> > > 
> > > Monday morning and the caffiene is just beginning to reach my brain,
> > > but this seems counter-intuitive to me.  I consider myself someone who
> > > _does_ want to keep in touch with the majority of the community, and
> > > breaking things into N additional mailing lists makes that harder, not
> > > easier.  I _do_ include core team updates, mascots, social meetings in
> > > that set of things to pay a little attention to here, especially
> > > around summit/PTG/Forum/etc times.
> > > 
> > > I've seen a couple of descriptions of who this proposal is not
> > > intended to address, who exactly is expected to benefit from more
> > > mailing lists?
> > > 
> > 
> > Thanks for your reply. The proposed change is meant to benefit
> > readers of openstack-dev who do not have access to, or ability with,
> > sup/procmail/sieve/IMAP/etc., but do want to be able to be able to keep
> > up with the general flow of discussion in openstack-dev. We had a room
> > full of 10 or so cross-project minded folks, and only 3 of us felt that
> > we could keep up with the discussion threads that we even care about, much
> > less those that we might care about, but don't have time to even evaluate.
> > 
> > The idea would simply be that those directly involved in a team wouldn't
> > mind subscribing to a few more ML's to get relevant information about
> > the day to day workings of a team, but that for most people openstack-dev
> > would be sufficient.
> > 
> > The response to the suggestion tells me that we don't have agreement that
> > there is a problem. Perhaps those of us in the SWG room at the time were
> > simply falling victim to a small sample size and anecdotal data.
> > 
> > So, I'll ask more generally: do you believe that the single openstack-dev
> > mailing list is working fine and we should change nothing? If not, what
> > problems has it created for you? 
> 
> As a person who sends a lot of process-driven email to this list,
> it is not working for my needs to communicate with others.
> 
> Over the past few cycles when I was the release PTL, I always had
> a couple of PTLs say there was too much email on this list for them
> to read, and that they had not read my instructions for managing
> releases. That resulted in us having to train folks at the last
> minute, remind them of deadlines, deal with them missing deadlines,
> and otherwise increased the release team's workload.
> 
> It is possible the situation will improve now that the automation
> work is mostly complete and we expect to see fewer significant
> changes in the release workflow. That still leaves quite a few
> people regularly surprised by deadlines, though.
> 

The problem above is really the krux of it. Whether or not you can keep
up with the mailing list can be an unknown, unknown. Even now, those
who can't actually handle the mailing list traffic are in fact likely
missing this thread about whether or not people can handle the mailing
list traffic (credit fungi for pointing out this irony to me on IRC).

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][swg] per-project "Business only" moderated mailing lists

[Clint Byrum]

Excerpts from Matthew Treinish's message of 2017-02-27 13:03:56 -0500:
> On Mon, Feb 27, 2017 at 06:18:10PM +0100, Thierry Carrez wrote:
> > > Dean Troyer wrote:
> > >> On Mon, Feb 27, 2017 at 3:31 AM, Clint Byrum  wrote:
> > >> This is not for users who only want to see some projects. That is a well
> > >> understood space and the mailman filtering does handle it. This is for
> > >> those who want to monitor the overall health of the community, address
> > >> issues with cross-project specs, or participate in so many projects it
> > >> makes little sense to spend time filtering.
> > > 
> > > Monday morning and the caffiene is just beginning to reach my brain,
> > > but this seems counter-intuitive to me.  I consider myself someone who
> > > _does_ want to keep in touch with the majority of the community, and
> > > breaking things into N additional mailing lists makes that harder, not
> > > easier.  I _do_ include core team updates, mascots, social meetings in
> > > that set of things to pay a little attention to here, especially
> > > around summit/PTG/Forum/etc times.
> 
> +1, I'm also someone who also tries to keep an eye on a lot of projects and
> cross project work and will find this a lot more difficult.
> 
> > > 
> > > I've seen a couple of descriptions of who this proposal is not
> > > intended to address, who exactly is expected to benefit from more
> > > mailing lists?
> > 
> > I'm not (yet) convinced that getting rid of 10% of ML messages (the ones
> > that would go to the -business lists) is worth the hassle of setting up
> > 50 new lists, have people subscribe to them, and have overworked PTL
> > moderate them...
> 
> I agree with this. (although TBH I don't think I can be convinced) I
> also don't think in practice it will even be close to 10% of the ML traffic
> being routed to the per project lists.
> 

To be clear, I estimate 10% of _threads_, not traffic. Most people can
mentally file a thread away by subject, even if their mail client can't.

> > 
> > Also from my experience moderating such a -business list (the
> > openstack-tc list) I can say that it takes significant effort to avoid
> > having general-interest discussions there (or to close them when they
> > start from an innocent thread). Over those 50+ -business mailing-lists
> > I'm pretty sure a few would diverge and use the convenience of isolated
> > discussions without "outsiders" potentially chiming in. And they would
> > be pretty hard to detect...
> > 
> 
> Another similar counter example is the dedicated openstack-qa list, which has
> been dead for a long time now. This was something that had similar issues,
> although it wasn't a moderated list. What ended up happening was that the
> discussions happening there were siloed and no one ever noticed anything being
> discussed there. So things had to be cross posted to get any attention.
> Discussions also ended up being duplicated between the 2 lists (like ttx said
> he ties to avoid via active moderation). Which is why we dissolved the
> openstack-qa list and just reintegrated the discussion back into 
> openstack-dev.
>

I obviously failed at stating this but I'll say it again: The business
lists would never be for discussions of anything. They're for informing
each other of facts pertaining to your project only.

I'm refraining from thinking up new solutions until we've agreed upon a
set of problems to address. Thanks for responding. :)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][swg] per-project "Business only" moderated mailing lists

[Clint Byrum]

Excerpts from Dean Troyer's message of 2017-02-27 09:32:09 -0600:
> On Mon, Feb 27, 2017 at 3:31 AM, Clint Byrum  wrote:
> > This is not for users who only want to see some projects. That is a well
> > understood space and the mailman filtering does handle it. This is for
> > those who want to monitor the overall health of the community, address
> > issues with cross-project specs, or participate in so many projects it
> > makes little sense to spend time filtering.
> 
> Monday morning and the caffiene is just beginning to reach my brain,
> but this seems counter-intuitive to me.  I consider myself someone who
> _does_ want to keep in touch with the majority of the community, and
> breaking things into N additional mailing lists makes that harder, not
> easier.  I _do_ include core team updates, mascots, social meetings in
> that set of things to pay a little attention to here, especially
> around summit/PTG/Forum/etc times.
> 
> I've seen a couple of descriptions of who this proposal is not
> intended to address, who exactly is expected to benefit from more
> mailing lists?
> 

Thanks for your reply. The proposed change is meant to benefit
readers of openstack-dev who do not have access to, or ability with,
sup/procmail/sieve/IMAP/etc., but do want to be able to be able to keep
up with the general flow of discussion in openstack-dev. We had a room
full of 10 or so cross-project minded folks, and only 3 of us felt that
we could keep up with the discussion threads that we even care about, much
less those that we might care about, but don't have time to even evaluate.

The idea would simply be that those directly involved in a team wouldn't
mind subscribing to a few more ML's to get relevant information about
the day to day workings of a team, but that for most people openstack-dev
would be sufficient.

The response to the suggestion tells me that we don't have agreement that
there is a problem. Perhaps those of us in the SWG room at the time were
simply falling victim to a small sample size and anecdotal data.

So, I'll ask more generally: do you believe that the single openstack-dev
mailing list is working fine and we should change nothing? If not, what
problems has it created for you? 

Let's refrain from making suggestions about solutions until we've agreed
on any problems (or the lack thereof, hopefully?)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][swg] per-project "Business only" moderated mailing lists

[Clint Byrum]

Excerpts from Luigi Toscano's message of 2017-02-27 03:02:45 -0500:
> 
> - Original Message -
> > Excerpts from Shamail Tahir's message of 2017-02-27 00:44:44 -0500:
> > > Hi Clint,
> > > 
> > > On Mon, Feb 27, 2017 at 12:25 AM, Clint Byrum  wrote:
> > > 
> > > > Excerpts from Matt Riedemann's message of 2017-02-26 19:48:50 -0600:
> > > > > On 2/26/2017 6:52 PM, Clint Byrum wrote:
> > > > > > During some productive discussions in the Stewardship Working Group
> > > > > > PTG
> > > > > > room, the subject of the mailing list came up. The usual questions
> > > > > > around whether or not we should have per-project lists came up and
> > > > > > the
> > > > > > reasons we don't were re-affirmed. To recap those reasons:
> > > > > >
> > > > > >   * Cross posting is the pits
> > > > > >   * People don't always know at the beginning of a thread that a
> > > > > > discussion will need to go wider, leading to silos and 
> > > > > > confusion.
> > > > > >
> > > > > > So we turned to ways to help reduce peoples' load while reading
> > > > > > e-mail,
> > > > > > since many (most?) tend to opt out of reading openstack-dev.
> > > > > >
> > > > > > There are a number of ways that we can help, including teaching
> > > > > > people
> > > > > > to have more efficient workflows and use specific mail reading tools
> > > > > > (don't worry, we're not adding an NNTP gateway.. yet). But one that
> > > > > > received positive feedback from the room was to have moderated
> > > > > > business-only mailing lists for each project.
> > > > > >
> > > > > > Basically, there are things that we _do_ know will not go wider when
> > > > > > the thread begins. Just running through the threads on the February
> > > > > > thread index, there are a few obvious classes:
> > > > > >
> > > > > >   * Mascots
> > > > > >   * Social meetups
> > > > > >   * Meeting logistics
> > > > > >   * Core team membership
> > > > > >
> > > >
> > > I'm curious as to how much of the traffic (such as the examples given)
> > > generates message fatigue on new users but I do appreciate that we are
> > > trying to find solutions to make it easier to enter into the mailing lists
> > > around OpenStack without having to resort to digests.
> > > 
> > 
> > I think it's worth analyzing it, if somebody has time. I do not. My wild
> > ass guess is between 1 and 5 percent of all messages, but probably more
> > like 5-10 percent of threads, as a lot of them are the shorter, less
> > interesting threads.
> > 
> > These seem like small numbers, but cognitive load is not linear and the
> > number of threads people end up reading varies whether or not they use
> > tags.
> 
> I share the feeling too that those messages are the minority of the total 
> amount of real discussions, so moving them away won't address the problem and 
> lead to more complication to the rest of the world.
> 

Not every solution is a force multiplier. Some things are just fine
tuning. And again, cognitive load is not linear. If you are in a single
project silo, sure, these are easy to tune out. If you are trying to
work cross-project, you have to spend more time on each thread, even if
it is obvious.

> > 
> > > > > There are likely others. The idea is that these messages would go into
> > > > > a
> > > > > > ${project}-busin...@lists.openstack.org. Said list would be 
> > > > > > moderated
> > > > by
> > > > > > a team of the PTL's choosing, and we would admonish moderators to
> > > > reject
> > > > > > soundly any threads not obviously single project business related.
> > > >
> > > In this approach, we could send messages that fall within the ${
> > > project}-busin...@lists.openstack.org to the dev ML as well.  This would
> > > allow people who want only the ${project}-business news to get the content
> > > without having to get all messages from the dev ML but at the same time
> > > allow threads to be available to both subscribers (dev and
> > > ${project}-business}.
&

Re: [openstack-dev] [all][swg] per-project "Business only" moderated mailing lists

[Clint Byrum]

Excerpts from Shamail Tahir's message of 2017-02-27 00:44:44 -0500:
> Hi Clint,
> 
> On Mon, Feb 27, 2017 at 12:25 AM, Clint Byrum  wrote:
> 
> > Excerpts from Matt Riedemann's message of 2017-02-26 19:48:50 -0600:
> > > On 2/26/2017 6:52 PM, Clint Byrum wrote:
> > > > During some productive discussions in the Stewardship Working Group PTG
> > > > room, the subject of the mailing list came up. The usual questions
> > > > around whether or not we should have per-project lists came up and the
> > > > reasons we don't were re-affirmed. To recap those reasons:
> > > >
> > > >   * Cross posting is the pits
> > > >   * People don't always know at the beginning of a thread that a
> > > > discussion will need to go wider, leading to silos and confusion.
> > > >
> > > > So we turned to ways to help reduce peoples' load while reading e-mail,
> > > > since many (most?) tend to opt out of reading openstack-dev.
> > > >
> > > > There are a number of ways that we can help, including teaching people
> > > > to have more efficient workflows and use specific mail reading tools
> > > > (don't worry, we're not adding an NNTP gateway.. yet). But one that
> > > > received positive feedback from the room was to have moderated
> > > > business-only mailing lists for each project.
> > > >
> > > > Basically, there are things that we _do_ know will not go wider when
> > > > the thread begins. Just running through the threads on the February
> > > > thread index, there are a few obvious classes:
> > > >
> > > >   * Mascots
> > > >   * Social meetups
> > > >   * Meeting logistics
> > > >   * Core team membership
> > > >
> >
> I'm curious as to how much of the traffic (such as the examples given)
> generates message fatigue on new users but I do appreciate that we are
> trying to find solutions to make it easier to enter into the mailing lists
> around OpenStack without having to resort to digests.
> 

I think it's worth analyzing it, if somebody has time. I do not. My wild
ass guess is between 1 and 5 percent of all messages, but probably more
like 5-10 percent of threads, as a lot of them are the shorter, less
interesting threads.

These seem like small numbers, but cognitive load is not linear and the
number of threads people end up reading varies whether or not they use
tags.

> > > There are likely others. The idea is that these messages would go into a
> > > > ${project}-busin...@lists.openstack.org. Said list would be moderated
> > by
> > > > a team of the PTL's choosing, and we would admonish moderators to
> > reject
> > > > soundly any threads not obviously single project business related.
> >
> In this approach, we could send messages that fall within the ${
> project}-busin...@lists.openstack.org to the dev ML as well.  This would
> allow people who want only the ${project}-business news to get the content
> without having to get all messages from the dev ML but at the same time
> allow threads to be available to both subscribers (dev and
> ${project}-business}.
> 
> I hope we still advocate for subscribing to the openstack-dev mailing list
> even if a contributor is only starting with a single project (and not
> interested in cross-project things) because it allows for people to see
> conversations they might have expertise in or find a new project they want
> to contribute to based on learning something new about it.
> 

Wow, I must have failed in my wording ,sorry about that, because you
got it 100% backwards. The idea is that everyone stays in openstack-dev
for _all_ discussions (single-project as well). Only the most mundane
but necessary emails go on per-project "business lists". So there would
be zero point in ever subscribing to the business lists without also
subscribing to openstack-dev, and likewise, republishing business lists
to openstack-dev would defeat the entire point.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][swg] per-project "Business only" moderated mailing lists

[Clint Byrum]

Excerpts from Matt Riedemann's message of 2017-02-26 19:48:50 -0600:
> On 2/26/2017 6:52 PM, Clint Byrum wrote:
> > During some productive discussions in the Stewardship Working Group PTG
> > room, the subject of the mailing list came up. The usual questions
> > around whether or not we should have per-project lists came up and the
> > reasons we don't were re-affirmed. To recap those reasons:
> >
> >   * Cross posting is the pits
> >   * People don't always know at the beginning of a thread that a
> > discussion will need to go wider, leading to silos and confusion.
> >
> > So we turned to ways to help reduce peoples' load while reading e-mail,
> > since many (most?) tend to opt out of reading openstack-dev.
> >
> > There are a number of ways that we can help, including teaching people
> > to have more efficient workflows and use specific mail reading tools
> > (don't worry, we're not adding an NNTP gateway.. yet). But one that
> > received positive feedback from the room was to have moderated
> > business-only mailing lists for each project.
> >
> > Basically, there are things that we _do_ know will not go wider when
> > the thread begins. Just running through the threads on the February
> > thread index, there are a few obvious classes:
> >
> >   * Mascots
> >   * Social meetups
> >   * Meeting logistics
> >   * Core team membership
> >
> > There are likely others. The idea is that these messages would go into a
> > ${project}-busin...@lists.openstack.org. Said list would be moderated by
> > a team of the PTL's choosing, and we would admonish moderators to reject
> > soundly any threads not obviously single project business related.
> >
> > Thoughts? If this sounds good, I'll go ahead and write up a spec.
> > (openstack-specs?)
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> So business as in like parliamentary procedure business and not business 
> as in synergizing growth for the bottom line in a global economy?
> 

Parliamentary procedure, as the original message suggests. Things that
are not likely to grow into something bigger.

> I care about more than just the nova project, so I also pay attention to 
> non-nova specific stuff in other projects, mostly neutron, cinder, 
> ironic, glance, keystone and QA. If I want to know about their business 
> (can we say businazz to jazz it up?) I not only need to be in the 
> openstack-dev list but also their project-specific list? It's all going 
> to filter to the same folders I already have, but then I have to be in 
> several more mailing lists.
> 
> Who is the actual audience for this change? Is it to get the mundane 
> every day project stuff out of the openstack-dev list so people don't 
> have to read it and if so, who are those people and why can't they just 
> filter or ignore threads they don't care about properly?
> 
> I'm a -1 on this unless I'm missing how it makes things much much better 
> somehow.
> 

This isn't for you, or for me.

For instance, I use a highly efficient mail workflow with offlineimap and
"sup" that lets me read the whole mailing list and aggressively filter
threads without losing my mind and while still retaining useful search
ability. I do not use folders other than putting anything with a List-Id
into "lists". I published a little bit of it here btw:

https://github.com/SpamapS/ferk (Firehose Email Reading Kit)

But I never finished publishing docs or useful templates for sup config.

And really, I don't expect every new participant in OpenStack to adopt
such a workflow. It's taken me at least 6 years to get it dialed in and
it's tuned to not only OpenStack but Debian and other smaller projects.

You have taken the folder approach, and that is a bit less complicated
to set up than sup+offlineimap, but still does require that you know
how to filter by tag. It also means that you are experiencing one of
the problems with cross posting whenever anybody adds a tag, as in
that setup each message is duplicated into each folder, or you have a
'first match' sieve and then tag order becomes significant. Either way,
you have to flip back and forth to read a thread. Or maybe somebody has
an answer? Nobody in the room at the SWG session had one.

Anyway, that's fine tuning for old-hats at managing openstack-dev. We
are still throwing hundreds of me

[openstack-dev] [all][swg] per-project "Business only" moderated mailing lists

[Clint Byrum]

During some productive discussions in the Stewardship Working Group PTG
room, the subject of the mailing list came up. The usual questions
around whether or not we should have per-project lists came up and the
reasons we don't were re-affirmed. To recap those reasons:

  * Cross posting is the pits
  * People don't always know at the beginning of a thread that a
discussion will need to go wider, leading to silos and confusion.

So we turned to ways to help reduce peoples' load while reading e-mail,
since many (most?) tend to opt out of reading openstack-dev.

There are a number of ways that we can help, including teaching people
to have more efficient workflows and use specific mail reading tools
(don't worry, we're not adding an NNTP gateway.. yet). But one that
received positive feedback from the room was to have moderated
business-only mailing lists for each project.

Basically, there are things that we _do_ know will not go wider when
the thread begins. Just running through the threads on the February
thread index, there are a few obvious classes:

  * Mascots
  * Social meetups
  * Meeting logistics
  * Core team membership

There are likely others. The idea is that these messages would go into a
${project}-busin...@lists.openstack.org. Said list would be moderated by
a team of the PTL's choosing, and we would admonish moderators to reject
soundly any threads not obviously single project business related.

Thoughts? If this sounds good, I'll go ahead and write up a spec.
(openstack-specs?)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone]PKI token VS Fernet token

[Clint Byrum]

Excerpts from Lance Bragstad's message of 2017-02-25 13:07:58 -0600:
> Since both token formats rebuild the authorization context at validation
> time, we can remove some revocation events that are no longer needed. This
> means we won't be storing as many revocation events on role removal from
> domains and projects. Instead we will only rely on the revocation API to
> invalidate tokens for cases like specific token revocation or password
> changes (the new design of validation does role assignment enforcement for
> us automatically). This should reduce the amount of data being replicated
> due to massive amounts of revocation events.
> 

I didn't know that the work to make role removal non-event based was
even started much less done. Cool.

> We do still have some more work to do on this front, but I can dig into it
> and see what's left.
> 

Indeed, the less revocation events, the better the Fernet story is
for scalability.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone]PKI token VS Fernet token

[Clint Byrum]

Excerpts from joehuang's message of 2017-02-25 04:09:45 +:
> Hello, Matt,
> 
> Thank you for your reply, just as what you mentioned, for the slow changed 
> data, aync. replication should work. My concerns is that the impact of 
> replication delay, for example (though it's quite low chance to happen):
> 
> 1) Add new user/group/role in RegionOne, before the new user/group/role are 
> replicated to RegionTwo, the new user begin to access RegionTwo service, then 
> because the data has not arrived yet, the user's request to RegionTwo may be 
> rejected for the token vaildation failed in local KeyStone.
> 

I think this is entirely acceptable. You can even check with your
monitoring system to find out what the current replication lag is to
each region, and notify the user of how long it may take.

> 2)In token revoke case. If we remove the user'role in RegionOne, the token in 
> RegionOne will be invalid immediately, but before the remove operation 
> replicated to the RegionTwo, the user can still use the token to access the 
> services in RegionTwo. Although it may last in very short interval.
> 
> Is there someone can evaluate the security risk is affordable or not.
> 

The simple answer is that the window between a revocation event being
created, and being ubiquitous, is whatever the maximum replication lag
is between regions. So if you usually have 5 seconds of replication lag,
it will be 5 seconds. If you have a really write-heavy day, and you
suddenly have 5 minutes of replication lag, it will be 5 minutes.

The complicated component is that in async replication, reducing
replication lag is expensive. You don't have many options here. Reducing
writes on the master is one of them, but that isn't easy! Another is
filtering out tables on slaves so that you only replicate the tables
that you will be reading. But if there are lots of replication events,
that doesn't help.

One decent option is to switch to semi-sync replication:

https://dev.mysql.com/doc/refman/5.7/en/replication-semisync.html

That will at least make sure your writes aren't acknowledged until the
binlogs have been transferred everywhere. But if your master can take
writes a lot faster than your slaves, you may never catch up applying , no 
matter
how fast the binlogs are transferred.

The key is to evaluate your requirements and think through these
solutions. Good luck! :)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] The end of OpenStack packages in Debian?

[Clint Byrum]

Excerpts from Thomas Goirand's message of 2017-02-21 00:50:35 +0100:
> On 02/19/2017 08:43 PM, Clint Byrum wrote:
> > Excerpts from Thomas Goirand's message of 2017-02-19 00:58:01 +0100:
> >> On 02/18/2017 07:59 AM, Clint Byrum wrote:
> >>> Indeed, DPMT uses all the worst choices for maintaining most of the
> >>> python module packages in Debian. However, something will need to be
> >>> done to spread the load of maintaining the essential libraries, and the
> >>> usual answer to that for Python libraries is DPMT.
> >>
> >> I wish the Python team was more like the Perl one, who really is a well
> >> functioning with a strong team spirit and commitment, with a sense of
> >> collective responsibility. It's far from being the case in the DPMT.
> >>
> >> Moving packages to the DPMT will not magically get you new maintainers.
> >> Even within the team, there's unfortunately *a lot* of strong package
> >> ownership.
> >>
> > 
> > Whatever the issues are with that team, there's a _mountain_ of packages
> > to maintain, and only one team whose charter is to maintain python
> > modules. So we're going to have to deal with the shortcomings of that
> > relationship, or find more OpenStack specific maintainers.
> 
> I think there's a misunderstanding here. What I wrote is that the DPMT
> will *not* maintain packages just because they are pushed to the team,
> you will need to find maintainers for them. So that's the last option of
> your last sentence above that would work. The only issue is, nobody
> cared so far...
> 

I believe your experience is not typical, and the DPMT will in fact
assist with some of the more boring aspects of maintaining those
packages. Are they going to chase new upstream versions? No. But they'll
help with transitions, policy updates, RC bugs, etc.

> > It's also important that the generic libraries
> > we maintain, like stevedore, remain up to date in Debian so they don't
> > fall out of favor with users. Nothing kills a library like old versions
> > breaking apps.
> 
> Stevedore is a very good example. It build-depends on oslotest (to run
> unit tests), which itself needs os-client-config, oslo.config, which
> itself ... no need to continue, once you need oslo.config, you need
> everything else. So to continue to package something like Stevedore, we
> need nearly the full stack. That's equivalent to maintaining all of
> OpenStack (as I wrote: the cherry on top of the cake is the services,
> the bulk work is the Python modules).
> 

We may have to agree to disagree on that. The libraries and clients
should be an order of magnitude simpler than the services to maintain.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [architecture][nova][neutron][cinder][ceilometer][ironic] PTG stuff -- Arch-WG nova-compute-api fact-gathering session Tuesday 10:30 Macon

[Clint Byrum]

Excerpts from Dmitry Tantsur's message of 2017-02-18 18:54:59 +0100:
> 2017-02-17 19:16 GMT+01:00 Clint Byrum :
> 
> > Hello, I'm looking forward to seeing many of you next week in Atlanta.
> > We're going to be working on Arch-WG topics all day Tuesday, and if
> > you'd like to join us for that in general, please add your topic here:
> >
> > https://etherpad.openstack.org/p/ptg-architecture-workgroup
> >
> > I specifically want to call out an important discussion session for one
> > of our active work streams, nova-compute-api:
> >
> > https://review.openstack.org/411527
> > https://review.openstack.org/43
> >
> > At this point, we've gotten a ton of information from various
> > contributors, and I want to thank everyone who commented on 411527 with
> > helpful data. I'll be compiling the data we have into some bullet points
> > which I intend to share on the projector in an etherpad[1], and then invite
> > the room to ensure the accuracy and completeness of what we have there.
> > I grabbed two 30-minute slots in Macon for Tuesday to do this, and I'd
> > like to invite anyone who has thoughts on how nova-compute interacts to
> > join us and participate. If you will not be able to attend, please read
> > the documents and comments in the reviews above and fill in any information
> > you think is missing on the etherpad[1] so we can address it there.
> >
> > [1] https://etherpad.openstack.org/p/arch-wg-nova-compute-api-ptg-pike
> 
> 
> https://ethercalc.openstack.org/Pike-PTG-Discussion-Rooms says you're in
> South Capital, not Macon. Who is right? :)
> 

OPS, you're right. Yes, please come to South Capital, not Macon. :)

> I can attend, but I don't really understand what this topic is about. Mind
> providing at TL;DR? Why exactly should we change something around
> nova-compute <-> ironic?
> 

It's ok, if you don't have time to read the referenced documents, I'll
try to summarize:

nova-compute is communicated with through a variety of poorly documented
or poorly encapsulated APIs. os-brick and os-vif do odd things with lock
files (I think!), ceilometer inspects files on disk and system level
monitors.

The idea is to define the entry points into nova-compute and more
clearly draw a line where nova-compute's responsibility begins and other
services authority ends. This is already well defined between
nova-compute and nova-conductor, but not so much with others.

My reason for wanting this is simplification of the interactions with the
compute node so OpenStack can be enhanced and debugged without reading
the code of nova-compute.

For Ironic, I believe there may be ways in which nova-compute imposes
a bit of an odd structure on the baremetal service, and I'm interested
in hearing Ironic developers' opinions on that. I may be totally off
base.

> >
> >
> > Once we have this data, I'll likely spend a small amount of time grabbing
> > people from
> > each relevant project team on Wednesday/Thursday to get a deeper
> > understanding of some
> > of the pieces that we talk about on Tuesday.
> >
> > From that, as a group we'll produce a detailed analysis of all the ways
> > nova-compute is interacted with today, and ongoing efforts to change
> > them. If you are interested in this please do raise your hand and come
> > to our meetings[2] as my time to work on this is limited, and the idea
> > for the Arch-WG isn't "Arch-WG solves OpenStack" but "Arch-WG provides
> > a structure by which teams can raise understanding of architecture."
> >
> > [2] https://wiki.openstack.org/wiki/Meetings/Arch-WG
> >
> > Once we've produced that analysis, which we intend to land as a document
> > in our arch-wg repository, we'll produce a set of specs in the appropriate
> > places (likely openstack-specs) for how to get it to where we want to
> > go.
> >
> > Also, speaking of the meeting -- Since we'll all be meeting on Tuesday
> > at the PTG, the meeting for next week is cancelled.
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Device tagging: rebuild config drive upon instance reboot to refresh metadata on it

[Clint Byrum]

Excerpts from Jay Pipes's message of 2017-02-20 10:00:06 -0500:
> On 02/17/2017 02:28 PM, Artom Lifshitz wrote:
> > Early on in the inception of device role tagging, it was decided that
> > it's acceptable that the device metadata on the config drive lags
> > behind the metadata API, as long as it eventually catches up, for
> > example when the instance is rebooted and we get a chance to
> > regenerate the config drive.
> >
> > So far this hasn't really been a problem because devices could only be
> > tagged at instance boot time, and the tags never changed. So the
> > config drive was pretty always much up to date.
> >
> > In Pike the tagged device attachment series of patches [1] will
> > hopefully merge, and we'll be in a situation where device tags can
> > change during instance uptime, which makes it that much more important
> > to regenerate the config drive whenever we get a chance.
> >
> > However, when the config drive is first generated, some of the
> > information stored in there is only available at instance boot time
> > and is not persisted anywhere, as far as I can tell. Specifically, the
> > injected_files and admin_pass parameters [2] are passed from the API
> > and are not stored anywhere.
> >
> > This creates a problem when we want to regenerated the config drive,
> > because the information that we're supposed to put in it is no longer
> > available to us.
> >
> > We could start persisting this information in instance_extra, for
> > example, and pulling it up when the config drive is regenerated. We
> > could even conceivably hack something to read the metadata files from
> > the "old" config drive before refreshing them with new information.
> > However, is that really worth it? I feel like saying "the config drive
> > is static, deal with it - if you want to up to date metadata, use the
> > API" is an equally, if not more, valid option.
> 
> Yeah, config drive should, IMHO, be static, readonly. If you want to 
> change device tags or other configuration data after boot, use a 
> configuration management system or something like etcd watches. I don't 
> think Nova should be responsible for this.

I tend to agree with you, and I personally wouldn't write apps that need
this. However, in the interest of understanding the desire to change this,
I think the scenario is this:

1) Servers are booted with {n_tagged_devices} and come up, actions happen
using automated thing that reads device tags and reacts accordingly.

2) A new device is added to the general configuration.

3) New servers configure themselves with the new devices automatically. But
existing servers do not have those device tags in their config drive. In
order to configure these, one would now have to write a fair amount of
orchestration to duplicate what already exists for new servers.

While I'm a big fan of the cattle approach (just delete those old
servers!) I don't think OpenStack is constrained enough to say that
this is always going to be efficient. And writing two paths for server
configuration feels like repeating yourself.

I don't have a perfect answer to this, but I don't think "just don't
do that" is sufficient as a response. We allowed the tags in config
drive. We have to deal with the unintended consequences of that decision.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Device tagging: rebuild config drive upon instance reboot to refresh metadata on it

[Clint Byrum]

Excerpts from Artom Lifshitz's message of 2017-02-20 08:23:09 -0500:
> Config drive over read-only NFS anyone?
> 
> 
> A shared filesystem so that both Nova and the guest can do IO on it at the
> same time is indeed the proper way to solve this. But I'm afraid of the
> ramifications in terms of live migrations and all other operations we can
> do on VMs...
> 

What makes anyone think this will perform better than the metadata
service?

If we can hand people an address that is NFS-capable, we can hand them
an HTTP(S) URL that doesn't have performance problems.

> 
> Michael
> 
> On Sun, Feb 19, 2017 at 6:12 AM, Steve Gordon  wrote:
> 
> > - Original Message -
> > > From: "Artom Lifshitz" 
> > > To: "OpenStack Development Mailing List (not for usage questions)" <
> > openstack-dev@lists.openstack.org>
> > > Sent: Saturday, February 18, 2017 8:11:10 AM
> > > Subject: Re: [openstack-dev] [nova] Device tagging: rebuild config drive
> > upon instance reboot to refresh metadata on
> > > it
> > >
> > > In reply to Michael:
> > >
> > > > We have had this discussion several times in the past for other
> > reasons.
> > > > The
> > > > reality is that some people will never deploy the metadata API, so I
> > feel
> > > > like we need a better solution than what we have now.
> > >
> > > Aha, that's definitely a good reason to continue making the config
> > > drive a first-class citizen.
> >
> > The other reason is that the metadata API as it stands isn't an option for
> > folks trying to do IPV6-only IIRC.
> >
> > -Steve
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] The end of OpenStack packages in Debian?

[Clint Byrum]

Excerpts from Thomas Goirand's message of 2017-02-19 00:58:01 +0100:
> On 02/18/2017 07:59 AM, Clint Byrum wrote:
> > Indeed, DPMT uses all the worst choices for maintaining most of the
> > python module packages in Debian. However, something will need to be
> > done to spread the load of maintaining the essential libraries, and the
> > usual answer to that for Python libraries is DPMT.
> 
> I wish the Python team was more like the Perl one, who really is a well
> functioning with a strong team spirit and commitment, with a sense of
> collective responsibility. It's far from being the case in the DPMT.
> 
> Moving packages to the DPMT will not magically get you new maintainers.
> Even within the team, there's unfortunately *a lot* of strong package
> ownership.
> 

Whatever the issues are with that team, there's a _mountain_ of packages
to maintain, and only one team whose charter is to maintain python
modules. So we're going to have to deal with the shortcomings of that
relationship, or find more OpenStack specific maintainers.

My main concern is making sure Debian users can interact with OpenStack.
It's truly unfortunate that we may not have OpenStack's core kept well
maintained in Debian (though I can't see any reason Ubuntu's developers
wouldn't help with that -- feel free to chime in Ubuntu folks). But it's
vital to OpenStack and Debian's interests that Debian users be able
to grab the pieces of the OpenStack SDK that they need to communicate
with OpenStack clouds. It's also important that the generic libraries
we maintain, like stevedore, remain up to date in Debian so they don't
fall out of favor with users. Nothing kills a library like old versions
breaking apps.

I'm not going to rush off and do anything now. I don't have time. But
it's now on my radar, and when I find a window for it to fall into, I'll
start looking at how best to transition these. In the mean time I'll
hold out hope that we'll have good news and you can continue your work.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev