Re: [Openstack-operators] neutron metadata service crumples under load

[Jonathan Proulx]

Thanks Simon,

That seems to be it, may find more as I test it but first pass adding
the cache_url got by 64 instance test case to laumch without crashing
the metadata service.

-Jon

On Wed, Oct 22, 2014 at 3:25 AM, Simon Pasquier spasqu...@mirantis.com wrote:
 Hello Jonathan,
 Have you seen this discussion on the openstack-dev [1] that discusses the
 bug 1361357 [2]?
 I have no idea if it is related to your issue but FWIW a fix backport in the
 latest Icehouse release introduced performance regression for the metadata
 agent.
 BR
 Simon

 [1]
 http://lists.openstack.org/pipermail/openstack-dev/2014-October/048916.html
 [2] https://bugs.launchpad.net/cloud-archive/+bug/1361357

 On Wed, Oct 22, 2014 at 2:33 AM, Jonathan Proulx j...@jonproulx.com wrote:

 Ah there's the log many instances of:

 2014-10-21 19:50:15.527 12931 INFO neutron.wsgi [-] 10.10.167.98 - -
 [21/Oct/2014 19:50:15] GET /openstack/2012-08-10 HTTP/1.1 500 343
 120.411705

 2014-10-21 19:50:15.528 12931 ERROR
 neutron.agent.metadata.namespace_proxy [-] Unexpected error.
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy Traceback (most recent call
 last):
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy   File

 /usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py,
 line 74, in __cal
 l__
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy req.body)
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy   File

 /usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py,
 line 105, in _pro
 xy_request
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy
 connection_type=UnixDomainHTTPConnection)
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy   File
 /usr/lib/python2.7/dist-packages/httplib2/__init__.py, line 1569, in
 request
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy (response, content) =
 self._request(conn, authority, uri, request_uri, method, body,
 headers, redi
 rections, cachekey)
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy   File
 /usr/lib/python2.7/dist-packages/httplib2/__init__.py, line 1316, in
 _request
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy (response, content) =
 self._conn_request(conn, request_uri, method, body, headers)
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy   File
 /usr/lib/python2.7/dist-packages/httplib2/__init__.py, line 1285, in
 _conn_request
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy response =
 conn.getresponse()
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy   File
 /usr/lib/python2.7/httplib.py, line 1045, in getresponse
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy response.begin()
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy   File
 /usr/lib/python2.7/httplib.py, line 409, in begin
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy version, status, reason =
 self._read_status()
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy   File
 /usr/lib/python2.7/httplib.py, line 373, in _read_status
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy raise BadStatusLine(line)
 2014-10-21 19:50:15.528 12931 TRACE
 neutron.agent.metadata.namespace_proxy BadStatusLine: ''
 2014-10-21 19:50:15.528 12931 TRACE neutron.agent.metadata.namespace_proxy

 On Tue, Oct 21, 2014 at 8:17 PM, Jonathan Proulx j...@jonproulx.com
 wrote:
  running Icehouse + Neutron ML2/OVS and network names spaces.
 
  Was running well unitl recently, most recent change was switching to
  Ceph RBD for ephemeral storage on the hypervisors (and glance). I
  suspect this of being relevant because it makes the instances launch
  much more quickly.
 
  I haven't classified the breaking point but launching 64 instances
  deterministically breaks the metadata agent.
 
  The service seems to be running on the controller, but is not
  listening in the network namespace.  It seems to require restarting
  both the dhcp-agent and the metadata agent  to get it to go again.
 
  Even in debug mode I get no errors in the logs.
 
  Anyone seen this?
 
  -Jon

 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack Storage Backend: Sheepdog, Ceph, or GlusterFS

[Jonathan Proulx]

I see Ceph as the most unified storage solution for OpenStack. We're
starting to use it for Cinder, and are currently using it for Glance
and Nova.  We haven't used Swift for the 2.5 years we've been running,
but since we have recently deployed Ceph for these other uses will do
plan on rolling out access to the objectstore, probably through a
Swift interface though that's currently TBD we do have a current use
case so it is near the top of our queue.

Ceph storage backend for ephemeral nova instances is something no one
else seems to have mentioned but we find it a huge help.  If you have
a RAW image in glance's ceph rbd backend and use the ceph rbd backend
for your nova instances these will be copy on write clones of the
glance image.  This makes for very fast instance startup and efficient
use of storage. Regardless of if you have the CoW stuff plumbed
together the rbd storage does permit easily live migration (even if
/var/lib/nova which holds the libvirt.xml definition of th instance is
not on shared storage)

As a caveat this copy on write image creation requires a patched
version of nova in Icehouse (which I've been running in production a
couple months).  These were meant to land in the released version of
Juno but I haven't yet personally verified that they have.

On the block storage side Ceph passed LVM (the default driver) in the
most recent user survey as the most used Cinder driver (see
http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014
).  This means that part is well exercised and if you do run into any
issues there should be plenty of people.

While I'm happy thus far with Ceph, I encourage you to consider your
needs for each component and be sure a single choice will fit for you.
This is especially true if you have geographic distribution.  Ceph's
synchronous replication can be an issue in that case I hear (It is not
my case so IDK) .

-Jon

On Fri, Nov 7, 2014 at 10:50 AM, Joe Topjian j...@topjian.net wrote:
 Gluster indeed provides both block and object storage.

 We use the Gluster Cinder driver in production and, short of some initial
 hiccups, it's been running great.

 I have no experience with Gluster's object storage support, though, so I
 can't give an opinion about it -- just confirmation that it exists. :)

 My personal opinion is to just use Swift for object storage. Sure you won't
 have the whole unified storage thing going on, but you'll get Swift. :)

 On Thu, Nov 6, 2014 at 5:23 PM, Jesse Pretorius jesse.pretor...@gmail.com
 wrote:

 On 6 November 2014 13:01, Hossein Zabolzadeh zabolza...@gmail.com wrote:

 Thanks for your opinion. But I am looking for the real difference
 between them...
 - Which one is better support in openstack?
 - Which one provides better unified storage backend for all openstack
 storage controllers(cinder, swift and glance)?


 I don't think that Gluster or Sheepdog provide a storage back-end capable
 of providing block storage (cinder) and object storage (swift) back-ends.
 Only Ceph provides a properly unified back-end for both. Ceph has also been
 supported for cinder for over two years - it's very mature. The Rados
 Gateway (Object Storage/Swift Interface) is much more recent, but I have yet
 to see problems with it - as long as you do acknowledge that it is not
 Swift.

 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] State of Juno in Production

[Jonathan Proulx]

Recently (4 weeks?) moved from Icehouse to Juno. It was pretty smooth
(neutron has been much more well behaved though I know that's not
relevant to you).

One negative difference I noticed, but haven't really dug into yet
since it's not a common pattern here:

If I schedule 20 instances in one API call I get conductor timeouts
and zero launches.  If I make many parallel scheduling calls for 20
instances each response is good and scaling out to several hundred
parallel launches is much faster and with out the neutron timeout
errors that plagued me since I switched over to quantum in Grizzly.

As I said I've not looked deeply at this so it may be a local config
issue rather than something systemic with Juno, but if it's an
important use case for you be sure to take a good look at it.

-Jon

On Tue, Feb 17, 2015 at 12:56 PM, Joe Topjian j...@topjian.net wrote:
 Nice - thanks, Jesse. :)

 On Tue, Feb 17, 2015 at 10:35 AM, Jesse Keating j...@bluebox.net wrote:

 On 2/17/15 8:46 AM, Joe Topjian wrote:


 The only issue I'm aware of is that live snapshotting is disabled. Has
 anyone re-enabled this and seen issues? What was the procedure to
 re-enable?


 We've re-enabled it. Live snapshots take more system resources, which
 meant I had to dial back down my Rally test to validate how it could
 perform.

 To re-enable it, we reverted the upstream commit that disabled it.


 https://github.com/blueboxgroup/nova/commit/fa3a9208ea366489410b4828bd20a74a571287a6

 Once that was clear, and we had an upgraded version of libvirt in place,
 live snapshots just happened.

 As for the rest of your mail, we're going from Havana to Juno, and we have
 neutron, so some of our experience won't necessarily apply to you.

 --
 -jlk

 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Example configs

[Jonathan Proulx]

Hi All,

One of the requests that's come up a few times around here has been
for 'real' config example.

During the PHL Ops Midccyle we finally mad ea place to put them:
https://github.com/osops/example-configs

And over the past couple days I pushed up the configs from MIT CSAIL's
deploy.  There's a delightful real world mess, but there you go.

This can also serve as a meta example for how to organize example
configs, so if anyone has comments on the structure of how I put them
up now is probably a good time to discuss so that when others (if
there are others who can do this) put their configs on line the format
is both good and consistent.

The sanitization process is hairy...

I rgrep'ed through for things like 'password','token','connection' (to
get the DB connection strings) and probably a couple other things but
missed 'auth_encryption_key' in the Heat config.  Thankfully we're not
yet big heat users so changing this wasn't a disaster, and also
reminded me to switch to using trusts rather than the more than
slightly scary password deferred_auth_method...

So I think this will get all the bits you need to redact (and quite a
bit more but still less than looking at every line of every file):

rgrep -e key -e token -e encryption -e password

(note there's still a glance-cache.conf:swift_store_key in my configs
but it's an unused default)

If anyone finds things I left your PLEASE let me know (preferably
directly), hopefully I won't spend the rest of my life switching keys
and passwords :)


-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Example configs

[Jonathan Proulx]

On Mon, Mar 16, 2015 at 12:33 PM, Caius Howcroft
caius.howcr...@gmail.com wrote:
 For what its worth all bloomberg's configs are open source (apart from
 things like ips, tokens and such) and in chef templates:
 https://github.com/bloomberg/chef-bcpc/tree/master/cookbooks/bcpc/templates/default

 thats what we run in production on several clusters at the moment,

Ah yes, I knew about that once but had forgotten.

I added a section to the OSOps example repository README.md where we
can collect links to other sources of examples and put that info in as
a starter.

Thanks,
-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops meetup

[Jonathan Proulx]

On Fri, Mar 13, 2015 at 4:56 PM, Subbu Allamaraju su...@subbu.org wrote:
 Regarding the discussion on tags, here is my take from the discussion on
 Monday.

 1. There is vehement agreement that having an integrated release consisting
 of a small “inner ring” set of service is desirable. There are 6-7 projects
 that a majority of operators deploy.

 2. There is a general concern that dropping the integrated release process
 entirely creates more operability hurdles than that exist today.

There was definite concern about co-gating, being sure there is a
defined set of core project releases that work together. I don't think
fixing these in an integrated release was seen as the only solution.

For example if Neutron is releasing versions foo, bar and baz that all
gate with Nova foo there's no pressing reason that nova need to make
three releases just to keep naming in sync (or vice versa if Nova
moves at higher velocity the Neutron)

The pressing need i s the ability to express that when the cogating
set changes  so it is easily discoverable what set is tested togather.
Keeping the core integrated may be the simplest solution but I don't
think it was the only option in the room.

 3. Many other projects are not seeing adoption for reasons such as lacking a
 minimum viable feature set, lack of operational feedback, concerns on
 scalability etc. Examples include designate, trove, zaqar, ceilometer etc.

 4. There is also a concern of whether tags make the tent too large and
 chaotic as any project can essentially claim to be an OpenStack project.
 This may leave it open for vendors and distro providers to define a product
 out of those OpenStack projects, eventually fragmenting the open source
 nature

 5. Operators can certainly help shape an MVP feature set.

there is a list of email volunteer on
https://etherpad.openstack.org/p/PHL-ops-tags to make up a ops-tags
working group to continue these discussions, so anyone who couldn't
make it to PHL and is interested in being part of that working group
should probably sign on on the etherpad.

also those of us who have already signed on should probably figure out
what that means in terms of email discussion (should it be her or on
the user-comittee list or somewhere else) and possibly IRC meetings or
what have you...

-Jon

 Subbu


 On Mar 13, 2015, at 12:29 PM, Daniel Comnea comnea.d...@gmail.com wrote:

 Tim,

 Are you aware of any other feedback etherpads for the other discussions?

 Cheers,
 Dani

 P.S for Nova Sean provided one of the best summaries.


 On Tue, Mar 10, 2015 at 10:04 AM, Tim Bell tim.b...@cern.ch wrote:

 I don't think there is any recording. The etherpads and summaries on
 superuser.openstack.org give a feeling for the discussions though and are an
 interesting read. The tags one looked particularly lively.

 Tim

  -Original Message-
  From: Adam Huffman [mailto:adam.huff...@gmail.com]
  Sent: 10 March 2015 10:50
  To: Daniel Comnea
  Cc: Tim Bell; openstack-operators@lists.openstack.org
  Subject: Re: [Openstack-operators] Ops meetup
 
  I was going to ask that too...
 
  Cheers,
  Adam
 
  On Tue, Mar 10, 2015 at 7:38 AM, Daniel Comnea comnea.d...@gmail.com
  wrote:
   Hi Tim,
  
   For those who can't be physically there, will be any sort of
   recordings/ output coming out from this sessions?
  
   Thanks,
   Dani
  
   On Sat, Mar 7, 2015 at 7:24 PM, Tim Bell tim.b...@cern.ch wrote:
  
  
  
   Great to see lots of input for the ops meetup nexts week. Feel free
   to add your items to the agenda
  
  
  
  
   Session Etherpads:
  
   Draft Agenda:
   http://lists.openstack.org/pipermail/openstack-operators/2015-Februar
   y/006268.html
  
  
  
   Monday
  
   https://etherpad.openstack.org/p/PHL-ops-OVS
  
   https://etherpad.openstack.org/p/PHL-ops-security
  
   https://etherpad.openstack.org/p/PHL-ops-app-eco-wg
  
   https://etherpad.openstack.org/p/PHL-ops-tools-wg
  
   https://etherpad.openstack.org/p/PHL-ops-large-deployments
  
   https://etherpad.openstack.org/p/PHL-ops-tags
  
   https://etherpad.openstack.org/p/PHL-ops-hardware
  
   https://etherpad.openstack.org/p/PHL-ops-arch-show-tell
  
  
  
   Tuesday
  
   https://etherpad.openstack.org/p/PHL-ops-rabbit-queue
  
   https://etherpad.openstack.org/p/PHL-ops-nova-feedback
  
   https://etherpad.openstack.org/p/PHL-ops-network-performance
  
   https://etherpad.openstack.org/p/PHL-ops-capacity-mgmt
  
   https://etherpad.openstack.org/p/PHL-ops-testing-interop
  
   https://etherpad.openstack.org/p/PHL-ops-burning-issues
  
   https://etherpad.openstack.org/p/PHL-ops-packaging
  
   https://etherpad.openstack.org/p/PHL-ops-telco
  
   https://etherpad.openstack.org/p/PHL-ops-arch-show-tell
  
  
  
   Feedback on the event  what you'd like to see in Vancouver:
  
   https://etherpad.openstack.org/p/PHL-ops-feedback
  
  
  
  
  
  
   ___
   OpenStack-operators mailing list
   

Re: [Openstack-operators] What are people using for configuration management? Puppet? Chef? Other?

[Jonathan Proulx]

I use Puppet because that's what the rest of my infrastructure was
built on and it worked.  When I was at the decision point (2-3yrs ago)
for a replacement to CFEngine2 for site wide configuration management
Puppet and Chef were the only serious contenders and seemed equivalent
enough, deciding factor was a time out bug in Chef's Ohai service when
dealing with our insanely large passwd and group files.

OpenStack specifically I think most people go with either what hey
have or what they know best. The biannual user survey track popularity
of config management systems in the community
http://superuser.openstack.org/articles/openstack-user-survey-insights-november-2014
top three all have a decent following and support.  In order they are
Puppet, Chef, and Ansible, for whatever that's worth.

There's alse some links to public repos of production config
management setups over here:
https://github.com/osops/example-configs/blob/master/README.md,
they're not really explained in any way but they are real not just
examples.

My view of the state of Puppet's openstack modules is they have wide
use and an active (but small) development community.  Its not perfect
but it's imperfections are generally well understood and the
feature/project coverage is pretty good.

HTH,
-Jon

On Thu, Mar 26, 2015 at 12:40 PM, Forrest Flagg fostro.fl...@gmail.com wrote:
 Hi all,

 Getting ready to install a Juno or Kilo cloud and was wondering what people
 are using for configuration management to deploy openstack.  Are you using
 Puppet, Chef, something else?  What was the decision process for making your
 choice?

 Thanks,

 Forrest

 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] expanding to 2nd location

[Jonathan Proulx]

On Mon, May 4, 2015 at 9:42 PM, Tom Fifield t...@openstack.org wrote:

 Do you need users to be able to see it as one cloud, with a single API
 endpoint?

Define need :)

As many of you know my cloud is a University system and researchers
are nothing if not lazy, in the best possible sense of course :)  So
having a single API and scheduler so users don't *need* to think about
placement while using AZs so that they can (as Tim mentions a little
further dorm the thread)  is very attractive.  Managing complexity is
also important since we about 1 FTE equivalent (split between two or
three actual humans) to manage our cloud.

For partially technical partially political reasons we will not have
the same IP networks available at the second location.  With a bit of
heavy lifting on my end I could probably change this, but if i did it
would mean all the L3 would need to be routes for one of the sites
(because $reasons trust me).  So given that users would need to pick
which network to use, which would in fact be picking which site to
launch in, which sounds like it would rather be a Region.

So  Joe's model where Region2 slaves off Region1 for Keystone and
Glance is looking like the best fit. We could force users to balance
across regions by splitting their quota using the non-unified quota
model to our advantage.

Though I still have a bit of reeading to do apparently since I'd
forgotten about the Architecture Design Guide Jesse pointed out
http://docs.openstack.org/arch-design/content/multi_site.html

Thanks all,
-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] expanding to 2nd location

[Jonathan Proulx]

Hi All,

We're about to expand our OpenStack Cloud to a second datacenter.
Anyone one have opinions they'd like to share as to what I would and
should be worrying about or how to structure this?  Should I be
thinking cells or regions (or maybe both)?  Any obvious or not so
obvious pitfalls I should try to avoid?

Current scale is about 75 hypervisors.  Running juno on Ubuntu 14.04
using Ceph for volume storage, ephemeral block devices, and image
storage (as well as object store).  Bulk data storage for most (but by
no means all) of our workloads is at the current location (not that
that matters I suppose).

Second location is about 150km away and we'll have 10G (at least)
between sites. The expansion will be approximately the same size as
the existing cloud maybe slightly larger and given site capacities the
new location is also more likely to be where any future grown goes.

Thanks,
-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] YVR Wednesday Ops: Work sessions

[Jonathan Proulx]

Hi All,

If you look at the Sched for Wednesday there's a substantial pile of
sessions just called Ops: Work session.

They are all actually about pretty specific things and typically run 3
in parallel so you have some choices to make, but it's not real easy
to see what those are.

I've lovingly hand crafted a list that shows what the actually are,
when and where they are and etherpad links where available.

Hopefully I didn't miss any!  Also if you see something interesting
double check the rooms especially as I'm notoriously bad at
transposing things :)

Of course if you're not at the summit you can still preload those pads
with your concerns!

Wednesday May 20, 2015
---
9:00am - 9:40am
---
Puppet Team
   * Room 217
   * https://etherpad.openstack.org/p/YVR-ops-puppet

---
9:00am - 10:30am
---
HPC Working Group
   * Room 218
   *  https://etherpad.openstack.org/p/YVR-ops-hpc

The Telco Working Group
   * East Bld, 2/3
   * https://etherpad.openstack.org/p/YVR-ops-telco
---
9:50am - 10:30am
---
Chef
   * Room 217
   * https://etherpad.openstack.org/p/YVR-ops-chef
Making Metal Easy (Ironic)
   * Room 218
   * etherpad? anyone?
---
11:00am - 11:40am
---
Ansible
   * Room 217
   * https://etherpad.openstack.org/p/YVR-ops-ansible
---
11:00am - 12:30pm
---
Monitoring  Tools WG
   * Room 216
   * https://etherpad.openstack.org/p/YVR-ops-tools
Ops Tags WG
   * Room 218
   * https://etherpad.openstack.org/p/YVR-ops-tags
---
11:50am - 12:30pm
---
Ceph
   * Room 217
   * https://etherpad.openstack.org/p/YVR-ops-ceph
---
1:50pm - 2:30pm
---
Tech Choices (eg is MongoDB OK?)
   * Room 218
   * https://etherpad.openstack.org/p/YVR-ops-tech-choices
---
1:50pm - 3:20pm
---
Large Deployments Team
   * Room 216
   * https://etherpad.openstack.org/p/YVR-ops-large-deployments
Burning Issues
   * Room 217
   * https://etherpad.openstack.org/p/YVR-ops-burning-issues
---
2:40pm - 3:20pm
---
CMDB
   * Room 218
   * https://etherpad.openstack.org/p/YVR-ops-cmdb
---
3:30pm - 4:10pm
---
OPs Docs
   * Room 217
   * https://etherpad.openstack.org/p/YVR-ops-docs
Data plane transitions
   * Room 218
   * https://etherpad.openstack.org/p/YVR-ops-data-plane-transitions
---
4:30pm - 6:00pm
---
Upgrades
   * Room 216
   * https://etherpad.openstack.org/p/YVR-ops-upgrades
Logging
   * Room 217
   * https://etherpad.openstack.org/p/YVR-ops-logging
Packaging
   * Room 218
   * https://etherpad.openstack.org/p/YVR-ops-packaging

Hope that helps,
-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Tags] Tags Team Repo our first tags!

[Jonathan Proulx]

On Fri, Jun 5, 2015 at 4:34 AM, Thierry Carrez thie...@openstack.org wrote:

 One option is to abandon the idea and converge to using the same
 concept. Another option is to rename that rich data (project
 operational metadata ?) to avoid the confusion of calling with same
 name what is essentially two different things. That will open the door
 to defining tags on top of it.

I guess I'm still not seeing why binary is a hard requirement, but...

This is an interesting alternative which may give both sides
everything they want.  If this is the way we go I think the same WG
should handle the rich metadata and binary tags.  Generating binary
tags out of structured data sources should be pretty trivial, if we
really need to have an other layer of abstraction here.

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [puppet] OpenStack Puppet Modules Usage Questions

[Jonathan Proulx]

On Mon, Jun 15, 2015 at 7:46 PM, Richard Raseley rich...@raseley.com wrote:
 As part of wrapping up the few remaining 'loose ends' in moving the Puppet
 modules under the big tent, we are pressing forward with deprecating the
 previously used 'puppet-openst...@puppetlabs.com' mailing list in favor of
 both the openstack-dev and openstack-operators lists with the '[puppet]'
 tag.

 Usage of the openstack-dev list seems pretty straight forward, but we wanted
 to confirm with the broader community that this list (openstack-operators)
 was the appropriate venue for Puppet and OpenStack related usage questions.

 Any objections to this model?

I think makes perfect sense especially as other ops related working
groups are starting to use this list in a similar [tagged] way.

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] what is the different in use Qcow2 or Raw in Ceph

[Jonathan Proulx]

On Thu, May 28, 2015 at 3:21 PM, Fox, Kevin M kevin@pnnl.gov wrote:
 I've experienced the opposite problem though. Downloading raw images and 
 uploading them to the cloud is very slow. Doing it through qcow2 allows them 
 to be compressed over the slow links. Ideally, the Ceph driver would take a 
 qcow2 and convert it to raw on glance ingest rather then at boot.

There is a blueprint for this which if I read correctly has landed in
kilo (I'm still on juno and haven't started kilo testing yet):

https://blueprints.launchpad.net/glance/+spec/basic-import-conversion

https://review.openstack.org/#/c/159129/

I hope that it's landed because I too am really looking forward to that feature

-Jon

 Thanks,
 Kevin
 
 From: Dmitry Borodaenko [dborodae...@mirantis.com]
 Sent: Thursday, May 28, 2015 12:10 PM
 To: David Medberry
 Cc: openstack-operators@lists.openstack.org
 Subject: Re: [Openstack-operators] what is the different in use Qcow2 or Raw 
 in Ceph

 David is right, Ceph implements volume snapshotting at the RBD level,
 not even RADOS level: whole 2 levels of abstraction above file system.
 It doesn't matter if it's XFS, BtrFS, Ext4, or VFAT (if Ceph supported
 VFAT): Ceph RBD takes care of it before individual chunks of an RBD
 volume are passed to RADOS as objects and get written into the file
 system as files by an OSD process.

 The reason Fuel documentation recommends to disable QCOW2 format for
 images is that RBD does not support QCOW2 disks directly, so Nova and
 Cinder have to _convert_ a QCOW2 image into RAW format before passing
 it to QEMU's RBD driver. This means that you end up downloading the
 QCOW2 image from Ceph to a nova-compute node (first full copy),
 converting it (second full copy), and uploading the resultant RAW
 image back to Ceph (third full copy) just to launch a VM or create a
 volume from an image.


 On Thu, May 28, 2015 at 8:33 AM, David Medberry openst...@medberry.net 
 wrote:
 yep. It's at the CEPH level (not the XFS level.)

 On Thu, May 28, 2015 at 8:40 AM, Stephen Cousins steve.cous...@maine.edu
 wrote:

 Hi David,

 So Ceph will use Copy-on-write even with XFS?

 Thanks,

 Steve

 On Thu, May 28, 2015 at 10:36 AM, David Medberry openst...@medberry.net
 wrote:

 This isn't remotely related to btrfs. It works fine with XFS. Not sure
 how that works in Fuel, never used it.

 On Thu, May 28, 2015 at 8:01 AM, Forrest Flagg fostro.fl...@gmail.com
 wrote:

 I'm also curious about this.  Here are some other pieces of information
 relevant to the discussion.  Maybe someone here can clear this up for me 
 as
 well.  The documentation for Fuel 6.0, not sure what they changed for 6.1,
 [1] states that when using Ceph one should disable qcow2 so that images 
 are
 stored in raw format.  This is due to the fact that Ceph includes its own
 mechanisms for copy-on-write and snapshots.  According to the Ceph
 documentation [2], this is true only when using a BTRFS file system, but 
 in
 Fuel 6.0 Ceph uses XFS which doesn't provide this functionality.  Also, 
 [2]
 recommends not using BTRFS for production as it isn't considered fully
 mature.  In addition, Fuel 6.0 [3] states that OpenStack with raw images
 doesn't support snapshotting.

 Given this, why does Fuel suggest not using qcow2 with Ceph?  How can
 Ceph be useful if snapshotting isn't an option with raw images and qcow2
 isn't recommended?  Are there other factors to take into consideration 
 that
 I'm missing?

 [1]
 https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2
 [2]
 http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/
 [3]
 https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug

 Thanks,

 Forrest

 On Thu, May 28, 2015 at 8:02 AM, David Medberry openst...@medberry.net
 wrote:

 and better explained here:
 http://ceph.com/docs/master/rbd/qemu-rbd/

 On Thu, May 28, 2015 at 6:02 AM, David Medberry
 openst...@medberry.net wrote:

 The primary difference is the ability for CEPH to make zero byte
 copies. When you use qcow2, ceph must actually create a complete copy
 instead of a zero byte copy as it cannot do its own copy-on-write tricks
 with a qcow2 image.

 So, yes, it will work fine with qcow2 images but it won't be as
 performant as it is with RAW. Also, it will actually use more of the 
 native
 underlying storage.

 This is also shown as an Important Note in the CEPH docs:
 http://ceph.com/docs/master/rbd/rbd-openstack/

 On Thu, May 28, 2015 at 4:12 AM, Shake Chen shake.c...@gmail.com
 wrote:

 Hi

 Now I try to use Fuel 6.1 deploy openstack Juno, use Ceph as cinder,
 nova and glance backend.

 In Fuel document suggest if use ceph, suggest use RAW format image.

 but if I upload qcow2 image, seem working well.

 what is the different use qcow2 and RAW in Ceph?


 --
 Shake Chen


 ___
 OpenStack-operators mailing list
 

Re: [Openstack-operators] what is the different in use Qcow2 or Raw in Ceph

[Jonathan Proulx]

On Thu, May 28, 2015 at 3:34 PM, Warren Wang war...@wangspeed.com wrote:
 Even though we're using Ceph as a backend, we still use qcow2 images as our
 golden images, since we still have a significant (maybe majority) number of
 users using true ephemeral disks. It would be nice if glance was clever
 enough to convert where appropriate.

you can use RBD as your ephemeral backend as well, we do.

this gets us very fast starts and efficient use of storage from RAW
since the ephemeral disk is just a CoW clone of the glance image.
We'd previously been using relatively slow local disk (7.2k SATA) and
our  ceph implementation (ssd for xfs journaling, 7.2k SAS for osds)
has better performance than that for most of our work loads.

Snapshotting instances still takes a long journey getting written to
local disk then pushed back up to glance, there's work to make proper
RBD snapshots directly but AFAIK this has a way to go.

-Jon


 Warren

 Warren

 On Thu, May 28, 2015 at 3:21 PM, Fox, Kevin M kevin@pnnl.gov wrote:

 I've experienced the opposite problem though. Downloading raw images and
 uploading them to the cloud is very slow. Doing it through qcow2 allows them
 to be compressed over the slow links. Ideally, the Ceph driver would take a
 qcow2 and convert it to raw on glance ingest rather then at boot.

 Thanks,
 Kevin
 
 From: Dmitry Borodaenko [dborodae...@mirantis.com]
 Sent: Thursday, May 28, 2015 12:10 PM
 To: David Medberry
 Cc: openstack-operators@lists.openstack.org
 Subject: Re: [Openstack-operators] what is the different in use Qcow2 or
 Raw in Ceph

 David is right, Ceph implements volume snapshotting at the RBD level,
 not even RADOS level: whole 2 levels of abstraction above file system.
 It doesn't matter if it's XFS, BtrFS, Ext4, or VFAT (if Ceph supported
 VFAT): Ceph RBD takes care of it before individual chunks of an RBD
 volume are passed to RADOS as objects and get written into the file
 system as files by an OSD process.

 The reason Fuel documentation recommends to disable QCOW2 format for
 images is that RBD does not support QCOW2 disks directly, so Nova and
 Cinder have to _convert_ a QCOW2 image into RAW format before passing
 it to QEMU's RBD driver. This means that you end up downloading the
 QCOW2 image from Ceph to a nova-compute node (first full copy),
 converting it (second full copy), and uploading the resultant RAW
 image back to Ceph (third full copy) just to launch a VM or create a
 volume from an image.


 On Thu, May 28, 2015 at 8:33 AM, David Medberry openst...@medberry.net
 wrote:
  yep. It's at the CEPH level (not the XFS level.)
 
  On Thu, May 28, 2015 at 8:40 AM, Stephen Cousins
  steve.cous...@maine.edu
  wrote:
 
  Hi David,
 
  So Ceph will use Copy-on-write even with XFS?
 
  Thanks,
 
  Steve
 
  On Thu, May 28, 2015 at 10:36 AM, David Medberry
  openst...@medberry.net
  wrote:
 
  This isn't remotely related to btrfs. It works fine with XFS. Not sure
  how that works in Fuel, never used it.
 
  On Thu, May 28, 2015 at 8:01 AM, Forrest Flagg
  fostro.fl...@gmail.com
  wrote:
 
  I'm also curious about this.  Here are some other pieces of
  information
  relevant to the discussion.  Maybe someone here can clear this up for
  me as
  well.  The documentation for Fuel 6.0, not sure what they changed for
  6.1,
  [1] states that when using Ceph one should disable qcow2 so that
  images are
  stored in raw format.  This is due to the fact that Ceph includes its
  own
  mechanisms for copy-on-write and snapshots.  According to the Ceph
  documentation [2], this is true only when using a BTRFS file system,
  but in
  Fuel 6.0 Ceph uses XFS which doesn't provide this functionality.
  Also, [2]
  recommends not using BTRFS for production as it isn't considered
  fully
  mature.  In addition, Fuel 6.0 [3] states that OpenStack with raw
  images
  doesn't support snapshotting.
 
  Given this, why does Fuel suggest not using qcow2 with Ceph?  How can
  Ceph be useful if snapshotting isn't an option with raw images and
  qcow2
  isn't recommended?  Are there other factors to take into
  consideration that
  I'm missing?
 
  [1]
 
  https://docs.mirantis.com/openstack/fuel/fuel-6.0/terminology.html#qcow2
  [2]
 
  http://ceph.com/docs/master/rados/configuration/filesystem-recommendations/
  [3]
 
  https://docs.mirantis.com/openstack/fuel/fuel-6.0/user-guide.html#qcow-format-ug
 
  Thanks,
 
  Forrest
 
  On Thu, May 28, 2015 at 8:02 AM, David Medberry
  openst...@medberry.net
  wrote:
 
  and better explained here:
  http://ceph.com/docs/master/rbd/qemu-rbd/
 
  On Thu, May 28, 2015 at 6:02 AM, David Medberry
  openst...@medberry.net wrote:
 
  The primary difference is the ability for CEPH to make zero byte
  copies. When you use qcow2, ceph must actually create a complete
  copy
  instead of a zero byte copy as it cannot do its own copy-on-write
  tricks
  with a qcow2 image.
 
  So, yes, it will work fine with qcow2 images but it 

Re: [Openstack-operators] Scaling the Ops Meetup

[Jonathan Proulx]

On Wed, Jul 1, 2015 at 3:29 AM, Tom Fifield t...@openstack.org wrote:
 Team,

 It's great to see so much passion! :)

 Here's an attempt at a summary email. I'll wait until a later email to
 wade into the discussion myself ;) Feel free to jump in on any point.

 =Things we tend to agree on=

snip I agree on all those too.

 =Things still under discussion=
 Sell Tickets
 * Many people agreed that some moderate form of ticketing could be OK,
 but the question remains to what extent this should be priced (low
 fee? $100-200? cover costs?). A strong counterpoint was that paid
 ticketing makes it less accessible (see spirit), prevents some local
 attendance, and is unfair to smaller operators, though others noted that
 it may be the only practical way to raise funds in the future.

I think everyone agrees this is best kept as low a barrier as possible.

It would be interesting to know per attendee costs to help assess what
kind of barrier it would be.  Obviously if we get some corporate
underwriting that meets the 'we all agree'  low impact desires that
would help minimize this and if it can be zero it should be.

 Break into Regional Events
 * A number of viewpoints, ranging from multiple regional events to
 one event only [maybe with a travel fund] to one event that moves
 around [maybe even outside USA] to make it in the centre of USA for
 easier travel on average.

I think breaking into regional events would seriously undermine the
utility of the event unless someone has a really clever idea how to
run 3 or 4 locations as a single distributed event so we can actually
gather and share ideas among all of them (I don't see how that would
work).

I am uncomfortable with the US-centric nature of the ops events even
though it's been terribly convenient for me.  I would suggest if we so
start rotating continents (which I'm in favor of) we try and keep it
opposite the summit locations so those least likely to make the summit
are most likely to make the mid cycle that way no region gets left too
far behind.


 Capping Numbers (inc. Limit Attendees per Company)
 * A lot of disagreement here. Many argued that any kind of cap or
 barrier to entry detracts from the accessibility of the event. Others
 put forth that too few restrictions could dilute the ops-heavy attendee
 base, and implied that large companies might send too many people.

I think it's best to try addressing this socially at first.  Make it
clear space is at a premium and encourage attendees to send the
minimum number of people necessary to cover the sessions.

Setting a hard limit is hard because I can imagine larger and more
complex sites may have a legitimate need to send more people due to
greater role specialization or other reasons.


 Multiple Tracks
 * To help deal with room size, we could split into multiple tracks. The
 ideal number of tracks is not clear at this stage.

I'm not even sure what I think is best here, but these are my thoughts:

More tracks makes it harder for small to medium size sites to cover.
Not saying we shouldn't expand parallelism but we should be cautious.

My site is a private university cloud with order of 100 hypervisors,
we're more or less happy to send 2 people to summits and one to mid
cycles, at least that's what I've gotten them to pay for in the past.
Obviously we don't come close to covering summits.  The dual track
(for one attendee) in PHL was OK and conflicts weren't too bad.

The obvious alternative if we need more sessions would be to go longer
and honestly I'm not keen on that either and would probably prefer
wider over longer.


-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Scaling the Ops Meetup

[Jonathan Proulx]

On Thu, Jul 2, 2015 at 2:26 PM, Jesse Keating j...@bluebox.net wrote:
 BoD, unless they feel the need to delegate, at which point then maybe an
 Operators committee. But I'd hate to see more committees created.

I feel like this may be a User Committee thing, which is an existing
committee and sort-of-kind-of how this started I think.  Granted
that's a bit of a shadowy cabal at this point but hopefully we're on a
path to a better place with that...

-Jon


 - jlk

 On Thu, Jul 2, 2015 at 11:23 AM, Matt Fischer m...@mattfischer.com wrote:

 Are you proposing an Operators committee or do you mean the OpenStack BoD?

 On Thu, Jul 2, 2015 at 12:15 PM, Jesse Keating j...@bluebox.net wrote:

 Honestly I'm fine with the elected board helping to make this decision.
 Folks that want to underwrite the event can submit a proposal to host, board
 picks from the submissions? Having a wide vote on it seems overkill to me.

 Open call for submissions, board votes. Is that unreasonable?


 - jlk

 On Thu, Jul 2, 2015 at 8:23 AM, Tom Fifield t...@openstack.org wrote:

 OK, so I'm just going to throw this one out there to re-stoke the
 discussion ...

 Venue selection process.

 At the moment, there's a few of us who work hard in the shadows to make
 the best choice we can from a range of generous offers :)

 In our brave new world, I think this should be a bit more open, what do
 you think?

 What kind of structure do we need to make the best decision?


 Regards,


 Tom


 On 01/07/15 15:29, Tom Fifield wrote:
  Team,
 
  It's great to see so much passion! :)
 
  Here's an attempt at a summary email. I'll wait until a later email to
  wade into the discussion myself ;) Feel free to jump in on any point.
 
  =Things we tend to agree on=
  Spirit of the event
  * The response most people had in common was that they didn't want to
  see vendor booths :) Several others noted the importance that the
  event
  should remain accessible and ensure there were no barriers to
  attendance, space for networking with others and sharing information
  about deployments without fear of vendor harassment.
 
  Multiple Sponsors
  * are OK, but they are more like underwriters who should be OK with
  only
  modest acknowledgement (see previous: no booths). Preference for
  operator sponsors. Several ways to recognise them possible.
 
  Current Schedule Format
  * It appeared like the current format is working well in general, but
  could do with minor tweaks.
 
 
  =Things still under discussion=
  Sell Tickets
  * Many people agreed that some moderate form of ticketing could be OK,
  but the question remains to what extent this should be priced (low
  fee? $100-200? cover costs?). A strong counterpoint was that paid
  ticketing makes it less accessible (see spirit), prevents some local
  attendance, and is unfair to smaller operators, though others noted
  that
  it may be the only practical way to raise funds in the future.
 
  Break into Regional Events
  * A number of viewpoints, ranging from multiple regional events to
  one event only [maybe with a travel fund] to one event that moves
  around [maybe even outside USA] to make it in the centre of USA for
  easier travel on average.
 
 
  Capping Numbers (inc. Limit Attendees per Company)
  * A lot of disagreement here. Many argued that any kind of cap or
  barrier to entry detracts from the accessibility of the event. Others
  put forth that too few restrictions could dilute the ops-heavy
  attendee
  base, and implied that large companies might send too many people.
 
 
  Multiple Tracks
  * To help deal with room size, we could split into multiple tracks.
  The
  ideal number of tracks is not clear at this stage.
 
  Evening Event
  * Several people said they found the PHL evening event uncomfortably
  packed, and suggested cancelling it on this basis, or on the basis of
  cost. Suggested alternate was posting a list of nearby venues.
 
  Lightening Talks
  * Have lightening talks, perhaps by renaming show and tell. More of
  them? Arranged differently? Unclear.
 
  =Ideas=
  * Video Recording - Might be worth a shot, starting small.
  * Travel Fund, Scholarship Fund, Slush Fund
  * Use Universities during the summer break for venues
 
  =Open Questions=
  * How will the number of attendees grow?
  * What are the costs involved in hosting one of these events?
  * Stuff about the summit - probably need a different thread for this
 
 
  Regards,
 
 
  Tom
 
 
 
 
  On 30/06/15 12:33, Tom Fifield wrote:
  Hi all,
 
  Right now, behind-the-scenes, we're working on getting a venue for
  next
  ops mid-cycle. It's taking a little longer than normal, but rest
  assured
  it is happening.
 
  Why is it so difficult? As you may have noticed, we're reaching the
  size
  of event where both physically and financially, only the largest
  organisations can host us.
 
  We thought we might get away with organising this one old-school with
  a
  single host and sponsor. Then, for the next, start a 

Re: [Openstack-operators] Image Based Backups

[Jonathan Proulx]

On Mon, Jun 29, 2015 at 1:52 PM, Brendan Johnson bjohn...@paragusit.com wrote:
 What are people using to perform image based backups of Windows and Linux VMs 
 in OpenStack?  I am using KVM as the hypervisor, Ceph for block storage and 
 Swift for object storage.  I know Cinder can backup volumes that are not in 
 use (at least in Juno, that may have changed in Kilo).   I am looking for an 
 image backup solution that allow the backup of online KVM VMs.  Integration 
 with OpenStack would be nice but isn't a must.

Well outside OpenStack it rather depends on your backend...

I'm not actually doing this so take it for what it's worth, but a
possible workflow in OpenStack would be to create a Cinder 'snapshot'
this does work live though anything you do live has some possibility
of inconsistency, you'll get a copy that will seem 'crashed' similar
to recovering from a sudden power failure.

Obviously snapshot is different than backup, so potentially you could
make the snapshot into a full volume, which would be a detached copy,
and use that as the backup source...

It's a thought any way,
-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] [stable][all] Keeping Juno "alive" for longer.

[Jonathan Proulx]

On Mon, Nov 09, 2015 at 07:18:16PM +, Jeremy Stanley wrote:
:On 2015-11-09 19:01:36 + (+), Tom Cameron wrote:
:[...]
:> What do the user/operator surveys say about the usage of older
:> releases? What portion of the user base is actually on releases
:> prior to Havana?
:
:The most recent OpenStack User Survey Report has an awesome trending
:analysis answering this exact question. See page 22 (labeled 21) of
:https://www.openstack.org/assets/survey/Public-User-Survey-Report.pdf
:for a very appropriate graph.

It's important to note that this is with the current pressure to keep
up with 6mo cadence.

It doesn't reflect howmany would prefer slower LTS style updates
because that's not an option.

That said I agree focusing on making single cycle upgrades work
soothly is probably effort better spent at this time and once we have
a good story on single cycle live upgrades think about how/if we
extend that to a multi cycle LTS.

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OPs Midcycle location discussion.

[Jonathan Proulx]

Let me restate the question a bit as I think I'm hearing two different
responses that may be getting conflated.

Option 1:  There's a single Ops Midcycle that shifts around and we
look at ways to increase remote participation. (obviously this doesn't
preclude other meetups)

Option 2: There are multiple Ops Meetups around midcycle (presumably
starting with North America, Asia, and Europe) and we look at ways of
coordinationg those re reduce duplication of effort any synthesis of
results.

I was advocating option 1 mostly because I think synthesis of option 2
is harder than stepping up preparation of etherpads before sessions
and review of them afterward is which is motly the level of remote
participation I'd envision in the first case (possibly also running
some email threads on any reccommendations that come out and seem
controvertial for any reason)

So far though seems the tide is runiing toward option 2, multiple
meet-ups. Though wee're still at a very small sample size.

-Jon


On Mon, Nov 16, 2015 at 10:50:52AM -0500, Jonathan Proulx wrote:
:Hi All,
:
:1st User Committee IRC meeting will be today at 19:00UTC on
:#openstack-meeting, we haven't exactly settled on an agenda yet but I
:hope to raise this issue the...
:
:It has been suggested that we make the February 15-16 European Ops
:Meetup in Manchester UK [1] the 'official' OPs Midcycle.  Previously
:all mid cycles have been US based.
:
:Personally I like the idea of broadening or geographic reach rather
:than staying concentrated in North America. I particularly like it
:being 'opposite' the summit location.
:
:This would likely trade off some depth of participation as fewer
:of the same people would be able to travel to all midcycles in person.
:
:Discuss...(also come by  #openstack-meeting at 19:00 UTC if you think
:this needs real time discussion)
:
:-Jon
:
:
:-- 
:
:1. 
http://www.eventbrite.com/e/european-openstack-operators-meetup-tickets-19405855436?aff=es2

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] OPs Midcycle location discussion.

[Jonathan Proulx]

Hi All,

1st User Committee IRC meeting will be today at 19:00UTC on
#openstack-meeting, we haven't exactly settled on an agenda yet but I
hope to raise this issue the...

It has been suggested that we make the February 15-16 European Ops
Meetup in Manchester UK [1] the 'official' OPs Midcycle.  Previously
all mid cycles have been US based.

Personally I like the idea of broadening or geographic reach rather
than staying concentrated in North America. I particularly like it
being 'opposite' the summit location.

This would likely trade off some depth of participation as fewer
of the same people would be able to travel to all midcycles in person.

Discuss...(also come by  #openstack-meeting at 19:00 UTC if you think
this needs real time discussion)

-Jon


-- 

1. 
http://www.eventbrite.com/e/european-openstack-operators-meetup-tickets-19405855436?aff=es2

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OPs Midcycle location discussion.

[Jonathan Proulx]

On Mon, Nov 16, 2015 at 10:37:59AM -0700, Matt Fischer wrote:
:I think that sticking with a singular official one is the plan. It's
:difficult enough for the foundation to line up sponsors/hosts etc for a
:single meet-up. 

Thanks for bring that up.

I was just wondering howmuch Foundation resources went into making
these go and if dividing that were even feasible.

Sounds like this is a strong argument for one 'official' midcycle

-Jon

:I also think that there are some US/Asia folks that will
:attend a midcycle in Europe and by also hosting a competing one locally you
:may reduce the attendance at the main one which defeats the purpose. Those
:midcycles work best when we have lots of different voices providing input.
:
:On Mon, Nov 16, 2015 at 10:06 AM, Jonathan Proulx <j...@csail.mit.edu> wrote:
:
:> On Mon, Nov 16, 2015 at 04:55:33PM +, Kruithof, Piet wrote:
:> :Sorry, late to the conversation and maybe missing a bit of context.
:> :
:> :How may regional meetings are we thinking?  2-3? Or more?
:>
:> My basic question was One or Many.
:>
:> If Many then that's a further question, but probably 3 (north america,
:> asia, europe)  or possibly 4 (+ south america)
:>
:> :
:> :Piet
:> :
:> :
:> :
:> :
:> :Piet Kruithof
:> :Sr UX Architect, HP Helion Cloud
:> :PTL, OpenStack UX project
:> :
:> :"For every complex problem, there is a solution that is simple, neat and
:> wrong.”
:> :
:> :H L Menken
:> :
:> :
:> :From: Matt Jarvis <matt.jar...@datacentred.co.uk matt.jar...@datacentred.co.uk>>
:> :Date: Monday, November 16, 2015 at 9:23 AM
:> :To: Jonathan Proulx <j...@csail.mit.edu<mailto:j...@csail.mit.edu>>
:> :Cc: "openstack-operators@lists.openstack.org openstack-operators@lists.openstack.org>" <
:> openstack-operators@lists.openstack.org openstack-operators@lists.openstack.org>>
:> :Subject: Re: [Openstack-operators] OPs Midcycle location discussion.
:> :
:> :+1 from me, although I am admittedly biased ;) Personally I think the
:> wider participation in the ops feedback loop can only be a positive thing,
:> and there are definitely different perspectives and concerns to be had from
:> European operators given the different commercial landscape. I'm sure the
:> same is also true for Asia.
:> :
:> :On 16 November 2015 at 15:50, Jonathan Proulx <j...@csail.mit.edu j...@csail.mit.edu>> wrote:
:> :Hi All,
:> :
:> :1st User Committee IRC meeting will be today at 19:00UTC on
:> :#openstack-meeting, we haven't exactly settled on an agenda yet but I
:> :hope to raise this issue the...
:> :
:> :It has been suggested that we make the February 15-16 European Ops
:> :Meetup in Manchester UK [1] the 'official' OPs Midcycle.  Previously
:> :all mid cycles have been US based.
:> :
:> :Personally I like the idea of broadening or geographic reach rather
:> :than staying concentrated in North America. I particularly like it
:> :being 'opposite' the summit location.
:> :
:> :This would likely trade off some depth of participation as fewer
:> :of the same people would be able to travel to all midcycles in person.
:> :
:> :Discuss...(also come by  #openstack-meeting at 19:00 UTC if you think
:> :this needs real time discussion)
:> :
:> :-Jon
:> :
:> :
:> :--
:> :
:> :1.
:> 
http://www.eventbrite.com/e/european-openstack-operators-meetup-tickets-19405855436?aff=es2
:> :
:> :___
:> :OpenStack-operators mailing list
:> :OpenStack-operators@lists.openstack.org OpenStack-operators@lists.openstack.org>
:> :http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:> :
:> :
:> :
:> :--
:> :Matt Jarvis
:> :Head of Cloud Computing
:> :DataCentred
:> :Office: (+44)0161 8703985
:> :Mobile: (+44)07983 725372
:> :Email: matt.jar...@datacentred.co.uk<mailto:matt.jar...@datacentred.co.uk
:> >
:> :Website: http://www.datacentred.co.uk
:> :
:> :DataCentred Limited registered in England and Wales no. 05611763
:>
:> --
:>
:> ___
:> OpenStack-operators mailing list
:> OpenStack-operators@lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:>

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OPs Midcycle location discussion.

[Jonathan Proulx]

On Mon, Nov 16, 2015 at 04:55:33PM +, Kruithof, Piet wrote:
:Sorry, late to the conversation and maybe missing a bit of context.
:
:How may regional meetings are we thinking?  2-3? Or more?

My basic question was One or Many.

If Many then that's a further question, but probably 3 (north america,
asia, europe)  or possibly 4 (+ south america)

:
:Piet
:
:
:
:
:Piet Kruithof
:Sr UX Architect, HP Helion Cloud
:PTL, OpenStack UX project
:
:"For every complex problem, there is a solution that is simple, neat and 
wrong.”
:
:H L Menken
:
:
:From: Matt Jarvis 
<matt.jar...@datacentred.co.uk<mailto:matt.jar...@datacentred.co.uk>>
:Date: Monday, November 16, 2015 at 9:23 AM
:To: Jonathan Proulx <j...@csail.mit.edu<mailto:j...@csail.mit.edu>>
:Cc: 
"openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>"
 
<openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>>
:Subject: Re: [Openstack-operators] OPs Midcycle location discussion.
:
:+1 from me, although I am admittedly biased ;) Personally I think the wider 
participation in the ops feedback loop can only be a positive thing, and there 
are definitely different perspectives and concerns to be had from European 
operators given the different commercial landscape. I'm sure the same is also 
true for Asia.
:
:On 16 November 2015 at 15:50, Jonathan Proulx 
<j...@csail.mit.edu<mailto:j...@csail.mit.edu>> wrote:
:Hi All,
:
:1st User Committee IRC meeting will be today at 19:00UTC on
:#openstack-meeting, we haven't exactly settled on an agenda yet but I
:hope to raise this issue the...
:
:It has been suggested that we make the February 15-16 European Ops
:Meetup in Manchester UK [1] the 'official' OPs Midcycle.  Previously
:all mid cycles have been US based.
:
:Personally I like the idea of broadening or geographic reach rather
:than staying concentrated in North America. I particularly like it
:being 'opposite' the summit location.
:
:This would likely trade off some depth of participation as fewer
:of the same people would be able to travel to all midcycles in person.
:
:Discuss...(also come by  #openstack-meeting at 19:00 UTC if you think
:this needs real time discussion)
:
:-Jon
:
:
:--
:
:1. 
http://www.eventbrite.com/e/european-openstack-operators-meetup-tickets-19405855436?aff=es2
:
:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org>
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:
:
:
:--
:Matt Jarvis
:Head of Cloud Computing
:DataCentred
:Office: (+44)0161 8703985
:Mobile: (+44)07983 725372
:Email: matt.jar...@datacentred.co.uk<mailto:matt.jar...@datacentred.co.uk>
:Website: http://www.datacentred.co.uk
:
:DataCentred Limited registered in England and Wales no. 05611763

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Juno and Kilo Interoperability

[Jonathan Proulx]

Juno clients with Kilo endpoints are fine.

We recently upgraded from Juno to Kilo and many of our clients
(including Horizon) are still Juno with no problems.

We also had a Kilo Horizon (for testing) running against Juno
endpoints prior to the upgrade and that seemed to work as well but it
wasn't thoroughly tested so can't recommend it without further
testing.

-Jon

On Wed, Aug 26, 2015 at 7:24 AM, Eren Türkay er...@skyatlas.com wrote:
 Hello operators,

 I am wondering if anyone is using different versions of Openstack in different
 sites.

 We have our first site which is Juno, and we are now having another site where
 we are planning to deploy Kilo. Does anyone have experience with different
 versions of installation? Particularly, our Horizon and other clients will be
 Juno, but they will talk to secondary site which is Kilo. Inferring from the
 release notes, Kilo API looks backward compatible with Juno, so I'm a little
 optimistic about it but still I'm not sure.

 Any help is appreciated,
 Eren

 --
 Eren Türkay, System Administrator
 https://skyatlas.com/ | +90 850 885 0357

 Yildiz Teknik Universitesi Davutpasa Kampusu
 Teknopark Bolgesi, D2 Blok No:107
 Esenler, Istanbul Pk.34220


 ___
 OpenStack-operators mailing list
 OpenStack-operators@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Milti-site Keystone & Galera

[Jonathan Proulx]

Hi All,

I'm pretty close to opening a second region in my cloud at a second
physical location.

The plan so far had been to only share keystone between the regions
(nova, glance, cinder etc would be distinct) and implement this by
using MariaDB with galera replication between sites with each site
having it's own gmcast_segment to minimize the long distance catter
plus a 3rd site with a galera arbitrator for the obvious reason.

Today I was warned against using this in a multi writer setup. I'd planned
 on one writer per physical location.

I had been under the impression this was the 'done thing' for
geographically sepperate regions, was I wrong? Should I replicate just
for DR and always pick a single possible remote write site?

site to site link is 2x10G (different physical paths), short link is
2.2ms average latency (2.1ms low, 2.5ms high over 250 packets) long
link shouldn't be much longer but isn't yet complete to test.

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Milti-site Keystone & Galera

[Jonathan Proulx]

Thanks Jay & Matt,

That's basically what I thought, so I'll keep thinking it :)

We're not replicating glance DB because images will be stored in
different local Ceph storage on each side so the images won't be
directly available.  We thought about moving back to a file back end
and rsync'ing but RBD gets us lots of fun things we want to keep
(quick start, copy on write thin cloned ephemeral storage etc...) so
decided to live with making our users copy images around.

-Jon



On Tue, Sep 8, 2015 at 5:00 PM, Jay Pipes <jaypi...@gmail.com> wrote:
> On 09/08/2015 04:44 PM, Jonathan Proulx wrote:
>>
>> Hi All,
>>
>> I'm pretty close to opening a second region in my cloud at a second
>> physical location.
>>
>> The plan so far had been to only share keystone between the regions
>> (nova, glance, cinder etc would be distinct) and implement this by
>> using MariaDB with galera replication between sites with each site
>> having it's own gmcast_segment to minimize the long distance catter
>> plus a 3rd site with a galera arbitrator for the obvious reason.
>
>
> I would also strongly consider adding the Glance registry database to the
> same cross-WAN Galera cluster. At AT, we had such a setup for Keystone and
> Glance registry databases at 10+ deployment zones across 6+ datacenters
> across the nation. Besides adjusting the latency timeout for the Galera
> settings, we made no other modifications to our
> internal-to-an-availability-zone Nova database Galera cluster settings.
>
> The Keystone and Glance registry databases have a virtually identical read
> and write data access pattern: small record/row size, small number of
> INSERTs, virtually no UPDATE and DELETE calls, and heavy SELECT operations
> on a small data set. This data access pattern is an ideal fit for a
> WAN-replicated Galera cluster.
>
>> Today I was warned against using this in a multi writer setup. I'd planned
>>   on one writer per physical location.
>
>
> I don't know who warned you about this, but it's not an issue in the real
> world. We ran in full multi-writer mode, with each deployment zone writing
> to and reading from its nearest Galera cluster nodes. No issues.
>
> Best,
> -jay
>
>> I had been under the impression this was the 'done thing' for
>> geographically sepperate regions, was I wrong? Should I replicate just
>> for DR and always pick a single possible remote write site?
>>
>> site to site link is 2x10G (different physical paths), short link is
>> 2.2ms average latency (2.1ms low, 2.5ms high over 250 packets) long
>> link shouldn't be much longer but isn't yet complete to test.
>>
>> -Jon
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] "Master" keystone and "sub" keystone

[Jonathan Proulx]

On Mon, Sep 28, 2015 at 03:31:54PM -0400, Adam Young wrote:
:On 09/26/2015 11:19 PM, RunnerCheng wrote:
:>Hi All,
:>I'm a newbie of keystone, and I'm doing some research about it
:>recently. I have a question about how to deploy it. The scenario is
:>on below:
:>
:>One comany has one headquarter dc and 5 sub dc locate in different
:>cities. We want to deploy separate OpenStack with "sub" keystone at
:>the sub dc, and want to deploy one "master" keystone at headquarter
:>dc. We want to manage all users, roles and tenants etc on the
:>"master" keystone, however we want the end-user can authenticate
:>with the "sub" keystone where he or she is locate.
:
:
:Use LDAP for the users, don't keep them in Keystone.
:
:Replicate roles, projects etc from master to sub.
:
:Use Fernet tokens.  Replicate revocation events both ways.

I'm hearing conflicting advice about the suitibility of Fernet tokens
for production use.

I like the idea. I did get them to work in kilo trivially for CLI, but
Horizon was unhappy for reasons I didn't fully investigate as I heard
they 'weren't quite ready in kilo' so I defered further investigation
to next cycle.

Though honestly if you're building somthing new right now starting
with Liberty is probably the right thing anyway by the time you're
done PoC it will be released.

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Meetups Team - Meeting Tuesday 1400 UTC (mid-cycle venue selection!)

[Jonathan Proulx]

This means <24hr to add your choice to the non binding doodle poll:

http://doodle.com/poll/e4heruzps4g94syf

Venue details recored here:

https://etherpad.openstack.org/p/ops-meetup-venue-discuss

And of course if you are available to join us tomorrow at 1400UTC
please do 

-Jon

On Mon, Jun 20, 2016 at 12:26:07PM +0800, Tom Fifield wrote:
:
:Hi all,
:
:We're having a special meeting:
:
:Tuesday, 21 of Jun at 1400 UTC [1]
:
:where we will decide the venue for the next mid-cycle will be!
:
:See you in IRC[2], in the #openstack-operators channel.
:
:
:Details about the group, and the link to the agenda etherpad, can be
:found at:
:
:https://wiki.openstack.org/wiki/Ops_Meetups_Team#Meeting_Information
:
:
:
:Regards,
:
:
:Tom
:
:
:[1] To see this in your local time - check: 
http://www.timeanddate.com/worldclock/fixedtime.html?msg=Ops+Meetups+Team=20160621T14
:
:[2] If you're new to IRC, there's a great guide here:
:http://docs.openstack.org/upstream-training/irc.html
:
:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] libvirt cpu type per instance

[Jonathan Proulx]

On Fri, Mar 04, 2016 at 11:52:33AM +0800, gustavo panizzo (gfa) wrote:
:On Thu, Mar 03, 2016 at 03:52:49PM -0500, Jonathan Proulx wrote:
:> 
:> I have a user who wants to specify their libvirt CPU type to restrict
:> performance because they're modeling embeded systems.
:> 
:> I seem to vaguely recall there is/was a way to specify this either in
:> the instance type or maybe even in the image metadata, but I can't
:> seem to find it.
:> 
:> Am I delusional or blind?


As previous posters pointed out I was a bit delusional obviously cpu
type only limits instruction set & is useful for bianry compatibility
but mothing to do with speed limiting or equalization. 

:you can use
:https://wiki.openstack.org/wiki/InstanceResourceQuota

Thanks I think that has the bits I'm looking for.  Had been reading on
a different document about cpu_shares (which are relative and thus not
what I want), but some how missed "cpu_quota" which may be the right
thing for me

so I was also a bit blind :)

Thanks,
-Jon


:
:to create "slow" flavors
:
:> 
:> -Jon 
:> 
:> -- 
:> 
:> ___
:> OpenStack-operators mailing list
:> OpenStack-operators@lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:
:-- 
:1AE0 322E B8F7 4717 BDEA BF1D 44BB 1BA7 9F6C 6333
:
:keybase: http://keybase.io/gfa

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-community] Recognising Ops contributions

[Jonathan Proulx]

On Thu, Mar 03, 2016 at 03:57:22PM +0100, Pierre Freund wrote:
:>
:> *This needs a catchy name.*
:> Yes, yes it does. Suggestions?
:>
:​​
:Some suggestions, but I'm not a native english speaker, it might sounds not
:natural.

As a native (american) english speaker all these suggestions sound
natural to me.

:AOC / Active Ops Contributor
:ACC / Active Community Contributor
:TOC / Technical Ops Contributor
:Proud Ops
:POP / Proudly Operating in Production
:IRO / I Run OpenStack


I was also thinking AOC since it mirrrors the existing ATC well.

ACC is also a good choice, though it implies a broader definition.  I
do think all active contibutors to the community should be
recognized.  It may be better to make the recoginitions more
specific.  For example ATC are also members of the community and are
clearly contributing so ACC could be seen to overlap with that.  If we
want to expand recognition beyond "Ops" perhaps we should add
desigantions?

-Jon



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Setting affinity based on instance type

[Jonathan Proulx]

On Wed, Mar 02, 2016 at 09:35:07PM -0500, Mathieu Gagné wrote:
:What would prevent the next user from having workloadB collocated with
:an other user's workloadA if that's the only capacity available?
:
:Unless aggregates are used, it will be hard to guaranty that workloadA
:and workloadB (from any users) are never collocated.
:
:You could probably play with custom weighers where a specialized
:aggregate would be preferred over the others unless there isn't capacity
:left. This would also mean that strict filters can't be used anymore
:like suggested. (and it will need custom Python code to be written)
:
:The main challenge I see is not the single first anti-affinity request,
:it's all the subsequent others which will also require anti-affinity.


My reading of the question suggesst they don't want a 'hard
never|always colocate' which hsot aggregates and server groups have
ways of enforcing, but rather a 'soft preference to avoid|achieve
colocation'.

I don't think there's an existing way to do this other than writing a
custom weighter.

I've frequntly wished for this scheduling option but not hard enough
to implement it myself...

-Jon

:
:Mathieu
:
:On 2016-03-02 8:46 PM, Adam Lawson wrote:
:> Hi Kris,
:> 
:> When using aggregates as an example, anyone can assign
:> workloadA<>aggregateA and workloadB<>aggregateB. That's easy. But if we
:> have outstanding requests for workloadB and have a glut of capacity in
:> aggregateA, workloadB won't be able to use those hosts so we have spare
:> capacity and no way to utilize it.
:> 
:> So I want to set an affinity for workloads and not at the host level.
:> That way, hosts remain fungible, workload affinity policies are
:> respected and cloud capacity is properly utilizing capacity.
:> 
:> Does that make sense?
:> 
:> //adam
:> 
:> */
:> Adam Lawson/*
:> 
:> AQORN, Inc.
:> 427 North Tatnall Street
:> Ste. 58461
:> Wilmington, Delaware 19801-2230
:> Toll-free: (844) 4-AQORN-NOW ext. 101
:> International: +1 302-387-4660
:> Direct: +1 916-246-2072
:> 
:> On Wed, Mar 2, 2016 at 3:08 PM, Kris G. Lindgren  > wrote:
:> 
:> You can set attributes on flavors that must match the attributes on
:> hosts or the host aggregates.  So you can basically always make sure
:> a specific flavors goes to a specific compute node or type (like
:> disks=ssd or class=gpu).  Look at nova flavor extra_specs
:> documentation and the aggregate_Instance_extra_specs under the
:> scheduler options.
:> 
:> 
:> ___
:> Kris Lindgren
:> Senior Linux Systems Engineer
:> GoDaddy
:> 
:> From: "Fox, Kevin M" >
:> Date: Wednesday, March 2, 2016 at 3:58 PM
:> To: Adam Lawson >,
:> "openstack-operators@lists.openstack.org
:> "
:>  >
:> Subject: Re: [Openstack-operators] Setting affinity based on
:> instance type
:> 
:> you usually do that on an instance level with server groups. do you
:> have an example where you might want to do it at the flavor level?
:> 
:> Thanks,
:> Kevin
:> 
:> *From:* Adam Lawson [alaw...@aqorn.com ]
:> *Sent:* Wednesday, March 02, 2016 2:48 PM
:> *To:* openstack-operators@lists.openstack.org
:> 
:> *Subject:* [Openstack-operators] Setting affinity based on instance type
:> 
:> I'm sure this is possible but I'm trying to find the info I need in
:> the docs so I figured I'd pitch this to you guys while I continue
:> looking:
:> 
:> Is it possible to set an affinity/anti-affinity policy to ensure
:> instance Type A is weighted for/against co-location on the same
:> physical host with instance Type B?
:> 
:> Basically I have no requirement for server-group affinity but rather
:> to ensure specific workloads are as separate as possible.
:> 
:> Thoughts?
:> 
:> //adam
:> 
:> */
:> Adam Lawson/*
:> 
:> AQORN, Inc.
:> 427 North Tatnall Street
:> Ste. 58461
:> Wilmington, Delaware 19801-2230
:> Toll-free: (844) 4-AQORN-NOW ext. 101
:> International: +1 302-387-4660 
:> Direct: +1 916-246-2072 
:> 
:> 
:> 
:> 
:> ___
:> OpenStack-operators mailing list
:> OpenStack-operators@lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:> 
:
:
:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org

[Openstack-operators] libvirt cpu type per instance?

[Jonathan Proulx]

I have a user who wants to specify their libvirt CPU type to restrict
performance because they're modeling embeded systems.

I seem to vaguely recall there is/was a way to specify this either in
the instance type or maybe even in the image metadata, but I can't
seem to find it.

Am I delusional or blind?

-Jon 

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-community] Recognising Ops contributions

[Jonathan Proulx]

On Fri, Mar 04, 2016 at 12:20:44PM +, Jeremy Stanley wrote:
:On 2016-03-04 10:02:36 +0100 (+0100), Thierry Carrez wrote:
:[...]
:> Upstream contributors are represented by the Technical Committee
:> and vote for it. Downstream contributors are represented by the
:> User Committee and (imho) should vote for it.
:[...]
:
:Right, this brings up the other important point I meant to make. The
:purpose of the "ATC" designation is to figure out who gets to vote
:for the Technical Committee, as a form of self-governance. That's
:all, but it's very important (in my opinion, far, far, far more
:important than some look-at-me status on a conference badge or a
:hand-out on free admission to an event). Granting votes for the
:upstream technical governing body to people who aren't involved
:directly in upstream technology decisions makes little sense, or at
:least causes it to cease being self-governance (as much as letting
:all of OpenStack's software developers decide who should run the
:User Committee would make it no longer well represent downstream
:users).

At the risk of drifting off topic that concern "letting all of
OpenStack's software developers decide who should run the User
Committee (UC)" is largely why the UC hasn't expanded to include
elected positions.

As currently written bylaws define the UC as 3 appointed positions. !
appointed by TC one by the board and the third by thte other two (FYI
I'm currently sitting in the TC apointed seat).  The by laws further
allow the UC to add seats elected by all foundation members.  In
Tokyo summit sessions where expantion was discussed the consensus was
to encourage more volunteer participation but not to add more formal
seats because there was no way to properly define the voting
constituency. Personally I can see both sides of that argument, but
the sense of the room was not to add elected positions untill we can
better deifne the constituency (that discussion could be reopened but
if you'd like to do so please start a new thread)

Perhaps nailing down this definition for recognition can actually have
broader implications and help to define who elects the UC.  It would
take a by-law change of course, but atleast we'd actually have a good
proposal (which we currently don't).

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] How are folks providing GPU instance types?

[Jonathan Proulx]

Hi All,

Having trouble finding any current info on best practices for
providing GPU instances.  Most of what Google is feeding me is Grizzly
or older.

I'm currently on Kilo (Mitaka upgrade planned in 60-90 days) with
Ubuntu14.04 and kvm hypervisor.  Looking to add some NVidia GPUs but
haven't invested in hardware yet.  Presumably I'd be using host
aggregates and instance metadata to separate these out from the
general pool so not tied to kvm though it would be nice to have image
compatibility across GPU and nonGPU instance types (this is currently
'raw' images in ceph rbd).

Any pointers to good info online or general advice as I travel down
this path?  I suppose particularly urgent is any hardware caveats I
need ot be aware of before I sink cash into the wrong thing (I'm
presuming that the k10,k20,k40,k80 are all equivalent in this regard).

-Jon

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] vmware to openstack

[Jonathan Proulx]

On Fri, May 06, 2016 at 11:52:38AM -0400, Silence Dogood wrote:
:PCI compliance / ITAR / TS stuff all require isolation.  You'd need to
:stand up isolated environments of the translation env for each.

Yes.  

I guess my point is isn't that the same compliance issue you have with
cloud in any case.  If the source and destination are sufficiently
isolated (which they must be since you are using them) and one of them
is OpenStack then this Coriolis migration service can run in that
OpenStack environment with the existing level of isolation. At least
that's what I get looking at their diagram

-Jon

:
:On Fri, May 6, 2016 at 11:50 AM, Jonathan Proulx <j...@csail.mit.edu> wrote:
:
:> On Fri, May 06, 2016 at 11:39:03AM -0400, Silence Dogood wrote:
:> :this strikes me as a really bad idea from a security standpoint... in fact
:> :it would violently violate like every audit / policy requirement I am
:> aware
:> :of.
:>
:> I come from a research environment with admittedly low thresholds for
:> this sort of thing (we don't do classified or medical stuff so no hard
:> regulations), but...
:>
:> If the same entity controls the source cloud, translation layer, and
:> destination cloud where's the harm?
:>
:> I can (maybe) see an issue if you're piping all you images through a
:> third part not involved in the source or target clouds. That's similar
:> to running on a public cloud trust wise so if you can trust a public
:> provider even that should be surmountable, but I don't think that's
:> the model suggested here.
:>
:> -Jon
:>
:> :
:> :-matt
:> :
:> :On Fri, May 6, 2016 at 11:32 AM, suresh kumar <boilingb...@gmail.com>
:> wrote:
:> :
:> :> Thanks Mariano, that really helps.
:> :>
:> :> On Fri, May 6, 2016 at 11:16 AM, Mariano Cunietti <mcunie...@enter.it>
:> :> wrote:
:> :>
:> :>>
:> :>> From: suresh kumar <boilingb...@gmail.com>
:> :>> Date: Friday 6 May 2016 at 17:07
:> :>> To: "openstack-operators@lists.openstack.org" <
:> :>> openstack-operators@lists.openstack.org>
:> :>> Subject: [Openstack-operators] vmware to openstack
:> :>>
:> :>>
:> :>> I am working on a project to migrate vm's from vmware to openstack
:> where
:> :>> am using tools to convert disk and later upload them to glance and spin
:> :>> instances are there any tools/scripts where I can migrate multiple
:> vm's at
:> :>> a time.
:> :>>
:> :>>
:> :>> https://cloudbase.it/cloud-migration-as-a-service/
:> :>>
:> :>>
:> :>
:> :> ___
:> :> OpenStack-operators mailing list
:> :> OpenStack-operators@lists.openstack.org
:> :> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:> :>
:> :>
:>
:> :___
:> :OpenStack-operators mailing list
:> :OpenStack-operators@lists.openstack.org
:> :http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:>
:>
:> --
:>

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] vmware to openstack

[Jonathan Proulx]

On Fri, May 06, 2016 at 11:39:03AM -0400, Silence Dogood wrote:
:this strikes me as a really bad idea from a security standpoint... in fact
:it would violently violate like every audit / policy requirement I am aware
:of.

I come from a research environment with admittedly low thresholds for
this sort of thing (we don't do classified or medical stuff so no hard
regulations), but...

If the same entity controls the source cloud, translation layer, and
destination cloud where's the harm?

I can (maybe) see an issue if you're piping all you images through a
third part not involved in the source or target clouds. That's similar
to running on a public cloud trust wise so if you can trust a public
provider even that should be surmountable, but I don't think that's
the model suggested here.

-Jon 

:
:-matt
:
:On Fri, May 6, 2016 at 11:32 AM, suresh kumar  wrote:
:
:> Thanks Mariano, that really helps.
:>
:> On Fri, May 6, 2016 at 11:16 AM, Mariano Cunietti 
:> wrote:
:>
:>>
:>> From: suresh kumar 
:>> Date: Friday 6 May 2016 at 17:07
:>> To: "openstack-operators@lists.openstack.org" <
:>> openstack-operators@lists.openstack.org>
:>> Subject: [Openstack-operators] vmware to openstack
:>>
:>>
:>> I am working on a project to migrate vm's from vmware to openstack where
:>> am using tools to convert disk and later upload them to glance and spin
:>> instances are there any tools/scripts where I can migrate multiple vm's at
:>> a time.
:>>
:>>
:>> https://cloudbase.it/cloud-migration-as-a-service/
:>>
:>>
:>
:> ___
:> OpenStack-operators mailing list
:> OpenStack-operators@lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:>
:>

:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Blazar? (Reservations and/or scheduled termination)

[Jonathan Proulx]

On Thu, Aug 04, 2016 at 09:59:15AM +0200, Álvaro López García wrote:

:For the record, we proposed a spec [1] on this long ago (actually, 1
:year ago).
:
:[1] https://review.openstack.org/#/c/104883/
:
:Your input is much welcome!

Thanks for the pointer, actually looks like you put in patch set one
on this over 2 years ago!

-Jon

:
:Cheers,
:-- 
:Álvaro López García  al...@ifca.unican.es
:Instituto de Física de Cantabria http://alvarolopez.github.io
:Ed. Juan Jordá, Campus UC  tel: (+34) 942 200 969
:Avda. de los Castros s/nskype: aloga.csic
:39005 Santander (SPAIN)



-- 


signature.asc
Description: Digital signature
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] PCI Passthrough issues

[Jonathan Proulx]

On Thu, Jul 07, 2016 at 11:13:29AM +1000, Blair Bethwaite wrote:
:Jon,
:
:Awesome, thanks for sharing. We've just run into an issue with SRIOV
:VF passthrough that sounds like it might be the same problem (device
:disappearing after a reboot), but haven't yet investigated deeply -
:this will help with somewhere to start!

:By the way, the nouveau mention was because we had missed it on some
:K80 hypervisors recently and seen passthrough apparently work, but
:then the NVIDIA drivers would not build in the guest as they claimed
:they could not find a supported device (despite the GPU being visible
:on the PCI bus). 

Definitely sage advice! 

:I have also heard passing mention of requiring qemu
:2.3+ but don't have any specific details of the related issue.

I didn't do a bisection but with qemu 2.2 (from ubuntu cloudarchive
kilo) I was sad and with 2.5 (from ubuntu cloudarchive mitaka but
installed on a kilo hypervisor) I am working.

Thanks,
-Jon


:Cheers,
:
:On 7 July 2016 at 08:13, Jonathan Proulx <j...@csail.mit.edu> wrote:
:> On Wed, Jul 06, 2016 at 12:32:26PM -0400, Jonathan D. Proulx wrote:
:> :
:> :I do have an odd remaining issue where I can run cuda jobs in the vm
:> :but snapshots fail and after pause (for snapshotting) the pci device
:> :can't be reattached (which is where i think it deletes the snapshot
:> :it took).  Got same issue with 3.16 and 4.4 kernels.
:> :
:> :Not very well categorized yet, but I'm hoping it's because the VM I
:> :was hacking on had it's libvirt.xml written out with the older qemu
:> :maybe?  It had been through a couple reboots of the physical system
:> :though.
:> :
:> :Currently building a fresh instance and bashing more keys...
:>
:> After an ugly bout of bashing I've solve my failing snapshot issue
:> which I'll post here in hopes of saving someonelse
:>
:> Short version:
:>
:> add "/dev/vfio/vfio rw," to  /etc/apparmor.d/abstractions/libvirt-qemu
:> add "ulimit -l unlimited" to /etc/init/libvirt-bin.conf
:>
:> Longer version:
:>
:> What was happening.
:>
:> * send snapshot request
:> * instance pauses while snapshot is pending
:> * instance attempt to resume
:> * fails to reattach pci device
:>   * nova-compute.log
:> Exception during message handling: internal error: unable to execute 
QEMU command 'device_add': Device initialization failedcompute.log
:>
:>   * qemu/.log
:> vfio: failed to open /dev/vfio/vfio: Permission denied
:> vfio: failed to setup container for group 48
:> vfio: failed to get group 48
:> * snapshot disappears
:> * instance resumes but without passed through device (hard reboot
:> reattaches)
:>
:> seeing permsission denied I though would be an easy fix but:
:>
:> # ls -l /dev/vfio/vfio
:> crw-rw-rw- 1 root root 10, 196 Jul  6 14:05 /dev/vfio/vfio
:>
:> so I'm guessing I'm in apparmor hell, I try adding "/dev/vfio/vfio
:> rw," to  /etc/apparmor.d/abstractions/libvirt-qemu rebooting the
:> hypervisor and trying again which gets me a different libvirt error
:> set:
:>
:> VFIO_MAP_DMA: -12
:> vfio_dma_map(0x5633a5fa69b0, 0x0, 0xa, 0x7f4e7be0) = -12 (Cannot 
allocate memory)
:>
:> kern.log (and thus dmesg) showing:
:> vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
:>
:> Getting rid of this one required inserting 'ulimit -l unlimited' into
:> /etc/init/libvirt-bin.conf in the 'script' section:
:>
:> 
:> script
:> [ -r /etc/default/libvirt-bin ] && . /etc/default/libvirt-bin
:> ulimit -l unlimited
:> exec /usr/sbin/libvirtd $libvirtd_opts
:> end script
:>
:>
:> -Jon
:>
:> ___
:> OpenStack-operators mailing list
:> OpenStack-operators@lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:
:
:
:-- 
:Cheers,
:~Blairo

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Neutron L2-GW operators experience [l2-gateway]

[Jonathan Proulx]

On Mon, Aug 15, 2016 at 05:33:13PM +0200, Saverio Proto wrote:

:I found that Ubuntu has packages here:
:http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/n/networking-l2gw/
:
:but I can't really get from the version number if these packages are
:supposed to be for Liberty or Mitaka.

:Has anyone on the list experience with this Neutron feature ?

I haven't but I can say the cloud archive packages are for Mitaka

Package is listed in:
http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/trusty-updates/mitaka/main/binary-amd64/Packages

But not in:
http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/trusty-updates/liberty/main/binary-amd64/Packages

-Jon

:
:thank you
:
:Saverio
:
:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] [openstack-dev] Large Contributing OpenStack Operators working group?

[Jonathan Proulx]

On Fri, Feb 03, 2017 at 04:34:20PM +0100, lebre.adr...@free.fr wrote:
:Hi, 
:
:I don't know whether there is already a concrete/effective way to identify 
overlapping between WGs. 
:But if not, one way can be to arrange one general session in each summit where 
all WG chairs could come and discuss about major actions that have been done 
for the past cycle and what are the plans for the next one.


That's a really good idea.  I think that woudl be a good use of the UC
Forum session.  In the past those had mostly been about what is the UC
and how shoudl it be structured going forward.  With recent by laws
change and upcoming ellection that's pretty settled.

Having a (very) brief report back from working groups and teams
followed by cross group discussion could be a valuable way forward for
that session IMO.

-Jon

:
:Being involved in several WGs allows us to identify collaboration 
opportunities (done for instance between the NFV and Massively Distributed 
WGs/Teams during this cycle) but to be honest it is costly and sometimes not 
still feasible to be involved in every action. 
:Offering the opportunity to get an up-to-date overview every 6 months can be 
valuable for all of us. 
:
:My two cents, 
:ad_rien_
:
:- Mail original -
:> De: "Jay Pipes" 
:> À: "Yih Leong Sun" , "Edgar Magana" 
,
:> openstack-operators@lists.openstack.org, user-commit...@lists.openstack.org
:> Cc: "JAMEY A MCCABE" , "ANDREW UKASICK" 
:> Envoyé: Vendredi 3 Février 2017 16:14:26
:> Objet: Re: [User-committee] [Openstack-operators] [openstack-dev] Large 
Contributing OpenStack Operators working
:> group?
:> 
:> Leong, thanks so much for responding. Comments/followup questions
:> inline.
:> 
:> On 02/02/2017 09:07 PM, Sun, Yih Leong wrote:
:> > LCOO was initiated by a group of large telco who contributes/uses
:> > OpenStack, such as AT, NTT, Reliance Jio, Orange, etc [1].
:> 
:> ack.
:> 
:> > The co-chair has reached out to Product WG for collaboration (refer
:> > IRC meeting logs). The team is working on plans to
:> > structure/define LCOO use cases.
:> 
:> My question here is what makes the LCOO use cases different from,
:> say,
:> the Telco Operator working group's use cases? Or the Massively
:> Distributed working group's use cases? Or the Enterprise working
:> group's
:> use cases?
:> 
:> Is the difference that the LCOO's use cases are stories that are
:> important for the LCOO member companies?
:> 
:> > Use case requirements (while still work-in-progress) can span
:> > across multiple areas which might/might-not covered by existing
:> > Team/WG.
:> 
:> Understood. Is the plan of the LCOO to identify use cases that are
:> covered by other working groups, contribute resources to develop that
:> use case, but have that existing working group handle the product
:> management (spec/blueprint/communication/roadmap) stuff?
:> 
:> > I'm sure LCOO will reach out to various projects for collaboration,
:> > stay tuned...
:> 
:> My questions seem to have been taken as an attack on the LCOO. I was
:> hoping to avoid that. I'm sincerely hoping to see the outreach to
:> various projects and am eager to collaborate with developers and
:> operators from the LCOO companies. I'm just confused what the
:> relationship between the LCOO and the existing working groups is.
:> 
:> Best,
:> -jay
:> 
:> > [1] https://etherpad.openstack.org/p/LCOO_Participants
:> >
:> > Thanks!
:> >
:> > ---
:> > Yih Leong Sun, PhD
:> > Senior Software Cloud Architect | Open Source Technology Center |
:> > Intel Corporation
:> > yih.leong@intel.com | +1 503 264 0610
:> >
:> >
:> > -Original Message-
:> > From: Jay Pipes [mailto:jaypi...@gmail.com]
:> > Sent: Thursday, February 2, 2017 5:23 PM
:> > To: Edgar Magana ;
:> > openstack-operators@lists.openstack.org;
:> > user-commit...@lists.openstack.org
:> > Cc: MCCABE, JAMEY A ; UKASICK, ANDREW
:> > 
:> > Subject: Re: [Openstack-operators] [openstack-dev] Large
:> > Contributing OpenStack Operators working group?
:> >
:> > On 02/02/2017 05:02 PM, Edgar Magana wrote:
:> >> Jay,
:> >>
:> >> I am including the WG chairs to make sure they answers your
:> >> questions and addresses your concerns.
:> >> In Barcelona the UC asked exactly the same questions and
:> >> recommended to the co-chairs of the LCOO WG to work with the
:> >> existing WG to identify overlapping activities and either to work
:> >> together or go ahead with the WG if there were not overlapping on
:> >> goals and deliverables.
:> >
:> > Was there any follow-on from that request from the UC?
:> >
:> >> I will let the co-chairs to follow up yours questions. BTW. I do
:> >> not think this topic should be posted in the openstack-dev
:> >> mailing list. So, I will BCC it.
:> >
:> > Sure, no problem.
:> >
:> >> Andrew and Jamey,
:> >>
:> >> Please, address these questions. Let’s work 

Re: [Openstack-operators] [User-committee] User Committee Election

[Jonathan Proulx]

Oops...Sorry Christopher

Seems to me that the transposed letters would not create any voter
confusion and since Christopher seems to agree and is the person I'd
see with standing to challenge I think we're OK to proceed.

-Jon

On Tue, Feb 14, 2017 at 04:40:58PM +, Matt Jarvis wrote:
:I have to apologise, this is entirely my fault and must have been picked up
:somewhere in a copy paste. Now the election is in progress, I can't
:actually update the candidates I'm afraid.
:
:Matt
:
:On Tue, Feb 14, 2017 at 4:33 PM, Edgar Magana 
:wrote:
:
:> Christopher,
:>
:> My apologies for the mistake. I also think that we could keep going if
:> fixing this makes things go weird in the election.
:>
:> Thanks for your support and good luck,
:>
:> Edgar
:>
:> On 2/13/17, 1:59 PM, "Christopher Aedo"  wrote:
:>
:> On Mon, Feb 13, 2017 at 9:06 AM, Shilla Saebi 
:> wrote:
:> > That information can be found at the below link:
:> > https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.
:> openstack.org_wiki_Governance_Foundation_UserCommittee_UC-
:> 2DElection-2DFeb17=DwIGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdO
:> ozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=JWEzNsO-
:> 4UxTvkMQ6CnegAmroVpIDJlC0T4CnaXoIgE=p3rrBZ3nirHmeBbeT6wzBBeG6Vyo9p
:> zqwpXJzpQVEbI=
:>
:> A friend just pointed out my name is misspelled on the ballot.  When I
:> voted this morning I completely overlooked that :)  Luckily it's
:> un-common enough to not be easily confused with any of the other
:> candidates.
:>
:> However, for the record, a vote for Christopher Adeo is a vote for
:> Christopher Aedo!  (Hopefully that will fly, rather than re-running
:> the election!)
:>
:> -Christopher
:>
:>
:> >
:> > On Mon, Feb 13, 2017 at 4:56 AM, Maish Saidel-Keesing <
:> mais...@maishsk.com>
:> > wrote:
:> >>
:> >> Hi Mark
:> >>
:> >>
:> >> On 13/02/17 11:42, Mark Baker wrote:
:> >>
:> >> Where can I see information about the candidates?
:> >>
:> >> Here are the Statements from each of the candidates.
:> >>
:> >>
:> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
:> openstack.org_pipermail_user-2Dcommittee_2017-2DFebruary_
:> 001652.html=DwIGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdO
:> ozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=JWEzNsO-
:> 4UxTvkMQ6CnegAmroVpIDJlC0T4CnaXoIgE=8dTN-1ldMm5FZ8K1s85gTVeU6EJ14_
:> HmGo57NVL9iFs=
:> >> - Christopher Aedo
:> >>
:> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
:> openstack.org_pipermail_user-2Dcommittee_2017-2DFebruary_
:> 001635.html=DwIGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdO
:> ozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=JWEzNsO-
:> 4UxTvkMQ6CnegAmroVpIDJlC0T4CnaXoIgE=npufNedOdtUphmTsWTodOwyBZWEfzy
:> 6Cy5TaGTU7nhg=
:> >> - Melvin Hillsman
:> >>
:> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
:> openstack.org_pipermail_user-2Dcommittee_2017-2DFebruary_
:> 001661.html=DwIGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdO
:> ozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=JWEzNsO-
:> 4UxTvkMQ6CnegAmroVpIDJlC0T4CnaXoIgE=4-YiEAsyxbdh_
:> dxsjUPPM3Ff2efJfOE2gbwTJqxyqbM=
:> >> - Shamail Tahir
:> >>
:> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
:> openstack.org_pipermail_user-2Dcommittee_2017-2DFebruary_
:> 001680.html=DwIGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdO
:> ozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=JWEzNsO-
:> 4UxTvkMQ6CnegAmroVpIDJlC0T4CnaXoIgE=QjoHeiqF1smBy4LsIxw9JUIKHX0_-
:> koOdIcbqwOmLAQ=
:> >> - Maish Saidel-Keesing
:> >>
:> >> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.
:> openstack.org_pipermail_user-2Dcommittee_2017-2DFebruary_
:> 001687.html=DwIGaQ=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdO
:> ozc=G0XRJfDQsuBvqa_wpWyDAUlSpeMV4W1qfWqBfctlWwQ=JWEzNsO-
:> 4UxTvkMQ6CnegAmroVpIDJlC0T4CnaXoIgE=TuwUf29F6YVHviMk0TG3uiL-
:> UDnivCqXxoAENsyQhfU=
:> >> - Yih Leong Sun
:> >>
:> >>
:> >>
:> >> Best Regards
:> >>
:> >>
:> >> Mark Baker
:> >>
:> >> On 11 February 2017 at 04:11, Matt Van Winkle <
:> mvanw...@rackspace.com>
:> >> wrote:
:> >>>
:> >>> Greetings!
:> >>>
:> >>>
:> >>>
:> >>> The time for our very first User Committee election is almost upon
:> us.
:> >>> In case you are unaware, there were recent changes – approved by
:> the Board
:> >>> of Directors – that allowed for the expansion of the UC from 3 to
:> 5 members
:> >>> and a move to elected positions.  Monday morning UTC, February
:> 13th,
:> >>> notifications will go out to all community members with the AUC
:> designation.
:> >>> The poll will stay open till 23:59 UTC on Friday, February 17th.
:> >>>
:> >>>
:> >>>
:> >>> This election, members will be voting on two additional seats for
:> the UC.
:> >>> We have a pool of five vetted 

Re: [Openstack-operators] What would you like in Pike?

[Jonathan Proulx]

What Tim said :)

my ordering:

1) Preemptable Instances -- this would be huge and life changing I'd
   give up any other improvements to get this.

2) Deeper utilization of nested projects -- mostly we find ways to
   mange with out this but it would be great to have.

   A) to allow research groups (our internal fiscal/administrative
   divisions) to sub divide quota allocations accordinf to their own
   priorities on a self serve basis (provided proper RBAC configs)
   B) to answer show back questions more easily.  Currently with flat
   projects individual research groups have multiple openstack
   projects, by convention we mange to usually be able to aggregaet
   them in reporting, but deing able to show susage by a parent and
   all it 's childeren woudl be very useful

3) Quota improvements -- this is important but we've learned to deal
   with it 

-Jon

On Sat, Jan 14, 2017 at 10:10:40AM +, Tim Bell wrote:
:There are a couple of items which have not been able to make it to the top 
priority for recent releases which would greatly simplify our day to day work 
with the users and make the cloud more flexible. The background use cases are 
described in 
https://openstack-in-production.blogspot.fr/2016/04/resource-management-at-cern.html
:
:
:-  Quota management improvements
:
:oManual interventions are often required to sync the current usage with 
the OpenStack view
:
:oNested projects are now in Keystone but is limited support in other 
projects for the end user benefit, such as delegation of quota for sub-projects
:
:-  Nova pre-emptible instances  
(https://review.openstack.org/#/c/104883/) to give spot market functionality
:
:oWe want to run our cloud at near 100% utilisation but this requires rapid 
ejection of lower priority VMs
:
:That having been said, I also fully support key priorities currently being 
worked on such as cells v2 and placement.
:
:Tim
:
:From: Melvin Hillsman 
:Date: Friday, 13 January 2017 at 02:30
:To: openstack-operators 
:Subject: [Openstack-operators] What would you like in Pike?
:
:Hey everyone,
:
:I am hoping to get a dialogue started to gain some insight around things 
Operators, Application Developers, and End Users would like to see happen in 
Pike. If you had a dedicated environment, dedicated team, and freedom to choose 
how you deployed, new features, older features, enhancements, etc, and were not 
required to deal with customer/client tickets, calls, and maintenances, could 
keep a good feedback loop between your team and the upstream community of any 
project, what would like to make happen or work on hoping the next release of 
OpenStack had/included/changed/enhanced/removed…?
:
:Kind regards,
:--
:Melvin Hillsman
:
:Ops Technical Lead
:OpenStack Innovation Center
:
:
:
:mrhills...@gmail.com
:phone: (210) 312-1267
:mobile: (210) 413-1659
:Learner | Ideation | Belief | Responsibility | Command
:
:http://osic.org
:
:

:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Keystone upgrade issues

[Jonathan Proulx]

Hi All,

working on testing our Kilo-> Mitaka keystone upgrade, and I've
clearly missied something I need to do or undo.

After DB migration and the edits I belive are required to paste and
conf files I can get tokens (using password auth) but it won't seem to
accept them (for example with an admin user I get 'action requires
authorization' errors when trying to show users )

Current setup is pretty simple and past upgrades of keystone have been
super easy, so other that reread and recheck not sure where I should
focus my attention.

using: 
fernet tokens 
mysql local users
apache/wsgi
Ubuntu 14.04 cloud archive packages 

This is what I can see with --debug the client (both
python-keystoneclient and python-openstackclient) after getting the
initial auth token through password exchange:

REQ: curl -g -i -X GET https://controller:35358/v2.0/users -H "User-Agent: 
python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: 
{SHA1}"
"GET /v2.0/users HTTP/1.1" 401 114
RESP: [401] Content-Length: 114 Vary: X-Auth-Token Keep-Alive: timeout=5 
Server: Apache/2.4.7 (Ubuntu) Connection: Keep-Alive Date: Thu, 25 Aug 2016 
14:41:26 GMT WWW-Authenticate: Keystone 
uri="https://nimbus.csail.mit.edu:35358; Content-Type: application/json 
X-Distribution: Ubuntu 
RESP BODY: {"error": {"message": "The request you have made requires 
authentication.", "code": 401, "title": "Unauthorized"}}

(v3 requests are similar modulo API differences)

Keysote.log in debug mode issues a couple deprecation warnings but no
errors (http://pastebin.com/WriB6u6i).  Not this log is for the same
event but response is UTC where log is local time (-0400)

Any pointer to where I should focus my investigations woudl be most
welcome :)

Thanks,
-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Keystone upgrade issues

[Jonathan Proulx]

On Thu, Aug 25, 2016 at 10:55:51AM -0400, Jonathan Proulx wrote:
:Hi All,
:
:working on testing our Kilo-> Mitaka keystone upgrade, and I've
:clearly missied something I need to do or undo.


D'Oh  why is it that public postings always lead me do discover my own
idiocy soon after (no matter howlong I've been staring at the
problem).

After most recent prodcution DB load into testing I forgot to switch
the endpoints to be in the test cluster.

mysql> update keystone.endpoint set 
url=replace(url,'production-endpoint','test-endpoint');

Obviously getting tokens from test and presenting them to production
endoints won't work :)

/me re-caffeinates

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] SDN for hybridcloud, does it *really* exist?

[Jonathan Proulx]

On Sat, Oct 01, 2016 at 02:39:38PM -0700, Clint Byrum wrote:

:I know it's hard to believe, but this world was foretold long ago and
:what you want requires no special equipment or changes to OpenStack,
:just will-power.  You can achieve it now if you can use operating system
:versions published in the last 5 or so years.
:
:The steps to do this:
:
:1) Fix your apps to work via IPv6
:2) Fix your internal users to have v6 native
:3) Attach your VMs and containers to a provider network with v6 subnets
:4) Use IPSec and firewalls for critical isolation. (What we use L2
:   separation for now)

That *is* hard to belive :) IPv6 has been coming soon since I started
in tech a very long time ago ... 

I will consider that but I have a diverse set of users I don't
control.  I *may* be able to apply pressure in the if you really need
this then do the right thing, but I probably still want a v4 solution
in my pocket.

-Jon
 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] SDN for hybridcloud, does it *really* exist?

[Jonathan Proulx]

So my sense from responses so far:

No one is doing unified SDN solutions across clouds and no one really
wants to.

Consensus is just treat each network island like another remote DC and
use normal VPN type stuff to glue them together.

( nod to http://romana.io an interesting looking network and security
automation project as a network agnostic alternative to SDN for
managing cross cloud policy on whatever networks are available. )

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] SDN for hybridcloud, does it *really* exist?

[Jonathan Proulx]

Starting to think refactoring my SDN world (currently just neutron
ml2/ovs inside OpenStack) in preparation for maybe finally lighting up
that second Region I've been threatening for the past year...

Networking is always the hardest design challeng.  Has anyone seen my
unicorn?  I dream of something the first works with neutron of course
but also can extend the same network features to hardware out side
openstack and into random public cloud infrastructures through VM and/or
containerised gateways.  Also I don't want to hire a whole networking
team to run it.

I'm fairly certain this is still fantasy though I've heard various
vendors promise the earth and stars but I'd love to hear if anyone is
actually getting close to this in production systems and if so what
your experience has been like.

-Jon

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Setting default MTU fro project networks?

[Jonathan Proulx]

On Mon, Nov 07, 2016 at 02:12:14PM -0800, Kevin Benton wrote:
:Which version of Neutron are you on now? Changing the config options had no
:impact on existing networks in Mitaka. After updating the config, only new
:networks will be affected. You will need to use an SQL query to update the
:existing network MTUs.

Mitaka

I understand that old MTU's won't change, but new overlays are gettign
created with 1458 MTU despite the configs I thnk should tell it the
jumbo underlay size, so I'm probably missing something :)

I did discover since neutron is now MTU aware I can simply drop the
dhcp-option=26,9000 and (after poking the DB for the existing jumbo
networks which had 'Null' MTUs) the old stuff and new stuff work just
new stuff has overly restrictive MTU.

:This was changed in Newton (https://review.openstack.org/#/c/336805/) but
:we couldn't back-port it because of the behavior change.

Neat I didn't know support form changing MTU was even planned, but I
gues it's here (well not quite *here* but...)

-Jon

:
:
:On Fri, Nov 4, 2016 at 10:34 AM, Jonathan Proulx <j...@csail.mit.edu> wrote:
:
:> Hi All,
:>
:>
:> So long story short how do I get my ml2/ovs GRE tenant network to default
:> to
:> MTU 9000 in Mitaka - or - get dhcp agents on on netork node to give
:> out different MTUs to different networks?
:>
:>
:> Seems between Kilo (my last release) and Mitaka (my  current production
:> world) Neutron got a lot cleverer about MTUs and teh simple
:> workarounds I had to make by jumbo frames go are now causing some
:> issues for newly created project networks.
:>
:> Because I'm setting 'dhcp-option=26,9000' in /etc/neutron/dnsmasq.conf
:> everything get an MTU of 9000 inside the guest OS. I only *really*
:> care about this for our provider vlans, for project networks I only
:> care that they work.
:>
:> CUrrently when a new project network is created it get an MTU of 1458
:> (1500 less GRE overhead) this is reflected in teh neutron DB and the
:> various virtual interfaces on the hypervisor and network node, but
:> DHCP configures inside the host to be 9000 and hilarity ensues.
:>
:> I tried setting DEFAULT/global_physnet_mtu=9134 in neutron.conf and
:> ml2/path_mtu=9134 in ml2.ini (which is the actual MTU of L2 links),
:> agent/veth_mtu=9134 was previously set. I thought this would result in
:> virtualdevices large enough to pass the 9000 traffic but seems to have
:> made no difference.
:>
:> I can kludge around by specifying MTU on network creation (or some
:> post facto DB hackery) but this isn't do able through my Horizon UI so
:> my users won't do it.
:>
:> Thanks,
:> -Jon
:>
:>
:> ___
:> OpenStack-operators mailing list
:> OpenStack-operators@lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:>

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] ML2/OVS odd GRE brokenness

[Jonathan Proulx]

I have an odd issue that seems to just be affecting one private
network for one tenant, though I saw a similar thing on a different
project network recently which I 'fixed' by rebooting the hypervisor.
Since this has now (maybe) happened twice I figure I should try to
understand what it is.

Given the following four VMs on 4 different hypervisors

vm1 on Hypervisor1
vm2 on Hypervisor2
vm3 on Hypervisor3
---
vm4 on Hypervisor4


vm1 -> vm3 talk fine among themselves but none to 4

examining ping traffic transiting from vm1-vm4 I can see arp requests
and responses at vm4 and GRE encapsulated ARP responses on
Hypervisor1's physical interface.

They look the same to me (same ecap id) coming in as the working vms
traffic, but they never make it to the qvo device which is before
iptables sec_group rules are applied at the tap device.

attempting to tare down and recreate this resuls in the same first 3
work last one doesn't split (possibly becuase scheduler puts them in
the same place? haven't checked) 

ovs-vsctl -- set Bridge br-int mirrors=@m  -- --id=@snooper2 get Port snooper2  
-- --id=@gre-801e0347 get Port gre-801e0347 -- --id=@m create Mirror 
name=mymirror select-dst-port=@gre-801e0347 select-src-port=@gre-801e0347 
output-port=@snooper2

tcpdump -i snooper2 

Only sees ARP requests but no response, what's broken if I can see GRE
encap ARP responses on physical interface but not on gre-
interface?  And why is it not broken for all tunnels endpoints?

Oddly if I boot a 5th VM on a 5th hypervisor it can talk to 4 but not 1-3 ...

hypervisors are Ubuntu 14.04 running Mitaka from cloud archive w/
xenial-lts kernels (4.4.0)

-Jon

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Setting default MTU for project networks?

[Jonathan Proulx]

On Mon, Nov 21, 2016 at 09:31:04PM +0100, Kostyantyn Volenbovskyi wrote:
:Hi,
:
:it really sounds that it path_mtu = 9000 was done in ‘wrong place’, see inline 
below 
:The fact that you indicate ml2.ini 
<https://www.google.com/search?client=safari=en=ml2.ini=UTF-8=UTF-8>
 and not ml2_conf.ini is a bit ‘suspicious'


Yes it is suspicious (and wrong) seems that [ml2] section is actually
in neutron/plugins/ml2/openvswitch_agent.ini so yes putting that in
the right place would probably fix things :)


:Reading the recent it sounds that in your exact case
:global_physnet_mtu should be equal to path_mtu and that should be 9000. That 
will result in MTU for VXLAN to be 8950 and MTU for VLAN - 9000
:
:So I don’t reason to use ‘9134’ or 9004 in your environment.

OK makes sense. So ignore the extra 4B for VLAN headers here.

Not that it matters but for the curious 9134 is what matches the
switch port MTU.  I no longer remember why but network folks had
initially (long before openstack) wanted a slightly larger than 9000
MTU for $REASONS. But in reality some hardware we needed on those
networks only supported 9000, so after seting the network hardware for
higher we turned back the MTU on all the stations to 9000


:All in all I think that [1] and [2] will be helpful.
:
:(and yes, it should be noted that my description is for Mitaka onwards)
:
:BR, 
:Konstantin
:
:[1] http://docs.openstack.org/draft/networking-guide/config-mtu.html 
<http://docs.openstack.org/draft/networking-guide/config-mtu.html>
:[2] https://bugzilla.redhat.com/show_bug.cgi?id=1374795 
<https://bugzilla.redhat.com/show_bug.cgi?id=1374795>
:
:> 
:> :Setting the following in your server config (not agent), should be enough
:> :for VXLAN networks to use a jumbo MTU.
:> :
:> :[DEFAULT]
:> :global_physnet_mtu = 9000
:> 
:> Got that one
:> 
:> :[ml2]
:> :path_mtu = 9000
:> 
:> So I don't have an [ml2] section in neutron.conf referenced by the
:> neutron-server processes.  I do have that in the agent.ini referenced
:> by neutron-openvswitch-agent on the network node.
:Well, typically you would have /etc/neutron/plugins/ml2/ml2_conf.ini, 
symlinked 
:from /etc/neutron/plugin.ini and neutron-server should use that.
:
:
:> I additonally have:
:> 
:> [agent]
:> veth_mtu=9004
:> 
:> in agent.ini on network node.
:> 
:> On hypervisors I have:
:> 
:> [ml2]
:> path_mtu = 0
:> physical_network_mtus =trunk:9004
:> 
:> [agent]
:> veth_mtu=9004
:> 
:> Obviously the hypervisor stuff won't effect how networks are created,
:> but don't want that to start biting me in a different way if I get
:> server side doing what I want.
:> 
:> 
:> Note 9004 is the pysical interface MTU in this example.  We have
:> provider netorks that are VLAN based so their MTU should be (is)
:> 9000.  The pre-existing provider networks are properly set though
:> manual hackery I've only added 1 since the cloud was initially created
:> 4 years ago, so not a common action.  Am I right in setting 9004 above
:> or should I still lie a little and provide the untagged MTU of 9000?
:> 
:> Thanks,
:> -Jon
:> 
:> :
:> :On Tue, Nov 8, 2016 at 8:31 AM, Jonathan Proulx <j...@csail.mit.edu 
<mailto:j...@csail.mit.edu>> wrote:
:> :
:> :> On Mon, Nov 07, 2016 at 02:12:14PM -0800, Kevin Benton wrote:
:> :> :Which version of Neutron are you on now? Changing the config options had
:> :> no
:> :> :impact on existing networks in Mitaka. After updating the config, only 
new
:> :> :networks will be affected. You will need to use an SQL query to update 
the
:> :> :existing network MTUs.
:> :>
:> :> Mitaka
:> :>
:> :> I understand that old MTU's won't change, but new overlays are gettign
:> :> created with 1458 MTU despite the configs I thnk should tell it the
:> :> jumbo underlay size, so I'm probably missing something :)
:> :>
:> :> I did discover since neutron is now MTU aware I can simply drop the
:> :> dhcp-option=26,9000 and (after poking the DB for the existing jumbo
:> :> networks which had 'Null' MTUs) the old stuff and new stuff work just
:> :> new stuff has overly restrictive MTU.
:> :>
:> :> :This was changed in Newton (https://review.openstack.org/#/c/336805/ 
<https://review.openstack.org/#/c/336805/>) but
:> :> :we couldn't back-port it because of the behavior change.
:> :>
:> :> Neat I didn't know support form changing MTU was even planned, but I
:> :> gues it's here (well not quite *here* but...)
:> :>
:> :> -Jon
:> :>
:> :> :
:> :> :
:> :> :On Fri, Nov 4, 2016 at 10:34 AM, Jonathan Proulx <j...@csail.mit.edu 
<mailto:j...@csail.mit.edu>>
:> :> wrote:
:> :> :
:> :> :> Hi All,
:> :> :>
:> 

Re: [Openstack-operators] need feedback about Glance image 'visibility' migration in Ocata

[Jonathan Proulx]

On Thu, Nov 17, 2016 at 01:27:39PM +, Brian Rosmaita wrote:
:On 11/17/16, 1:39 AM, "Sam Morrison" 
> wrote:
:
:On 17 Nov. 2016, at 3:49 pm, Brian Rosmaita 
> wrote:
:
:Ocata workflow:  (1) create an image with default visibility, (2) change
:its visibility to 'shared', (3) add image members
:
:Unsure why this can't be done in 2 steps, when someone adds an image member to 
a 'private' image the visibility changes to 'shared' automatically.
:Just seems an extra step for no reason?
:
:Thanks for asking, Sam, I'm sure others have the same question.
:
:Here's what we're thinking.  We want to avoid "magic" visibility transitions 
as a side effect of another action, and we want all means of changing 
visibility to be consistent going forward.  The two-step 1-1 sharing that 
automatically takes you from 'private' -> 'shared' is dangerous, as it can 
expose data and doesn't give an end user a way to make an image "really" 
private.  It's true that all an end user has to do under the new scheme is make 
one extra API call and then still shoot him/herself in the foot, but at least 
the end user has to remove the safety first by explicitly changing the 
visibility of the image from 'private' to 'shared' before the member-list has 
any effect.
:
:So basically, the reasons for the extra step are consistency and clarity.


The default 'shared' and migrate to 'shared' proposal make a lot of
sense to me as it keeps 99.99% of the functionality identical and the
one edge case described has prooper behaviour if a but opaque error
and is enduser solvable.


While having both 'private' and 'shared' images probably isn't a
distinction we'd use here, I can see the point.  It's like 'locking'
an instance, prevents accidental changes you don't want but is easily
switch off if you change your mind.  Presumably writing some clever
policy can even limit the number of people who could change from
'locked' to 'unlocked' in nova or 'private' to 'shared' in glance.

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Setting default MTU fro project networks?

[Jonathan Proulx]

Hi All,


So long story short how do I get my ml2/ovs GRE tenant network to default to
MTU 9000 in Mitaka - or - get dhcp agents on on netork node to give
out different MTUs to different networks?


Seems between Kilo (my last release) and Mitaka (my  current production
world) Neutron got a lot cleverer about MTUs and teh simple
workarounds I had to make by jumbo frames go are now causing some
issues for newly created project networks.

Because I'm setting 'dhcp-option=26,9000' in /etc/neutron/dnsmasq.conf
everything get an MTU of 9000 inside the guest OS. I only *really*
care about this for our provider vlans, for project networks I only
care that they work.

CUrrently when a new project network is created it get an MTU of 1458
(1500 less GRE overhead) this is reflected in teh neutron DB and the
various virtual interfaces on the hypervisor and network node, but
DHCP configures inside the host to be 9000 and hilarity ensues.

I tried setting DEFAULT/global_physnet_mtu=9134 in neutron.conf and
ml2/path_mtu=9134 in ml2.ini (which is the actual MTU of L2 links),
agent/veth_mtu=9134 was previously set. I thought this would result in
virtualdevices large enough to pass the 9000 traffic but seems to have
made no difference.

I can kludge around by specifying MTU on network creation (or some
post facto DB hackery) but this isn't do able through my Horizon UI so
my users won't do it.

Thanks,
-Jon


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] How do you even test for that?

[Jonathan Proulx]

Hi All,

Just on the other side of a Kilo->Mitaka upgrade (with a very brief
transit through Liberty in the middle).

As usual I've caught a few problems in production that I have no idea
how I could possibly have tested for because they relate to older
running instances and some remnants of older package versions on the
production side which wouldn't have existed in test unless I'd
installed the test server with Havana and done incremental upgrades
starting a fairly wide suite of test instances along the way.

First thing that bit me was neutron-db-manage being confused because
my production system still had migrations from Havana hanging around.
I'm calling this a packaging bug
https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1633576 but I
also feel like remembering release names forever might be a good
thing.

Later I discovered during the Juno release (maybe earlier ones too)
making snapshot of running instances populated the snapshot's meta
data with "instance_type_vcpu_weight: none".  Currently (Mitaka) this
value must be an integer if it is set or boot fails.  This has the
interesting side effect of putting your instance into shutdown/error
state if you try a hard reboot of a formerly working instance.  I
'fixed' this manually frobbing the DB to set lines where
instance_type_vcpu_weight was set to none to be deleted.

Does anyone have strategies on how to actually test for problems with
"old" artifacts like these?

Yes having things running from 18-24month old snapshots is "bad" and
yes it would be cleaner to install a fresh control plane at each
upgrade and cut over rather than doing an actual in place upgrade.  But
neither of these sub-optimal patterns are going all the way away
anytime soon.

-Jon

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Any serious stability and performance issues on RBD as ephemeral storage ?

[Jonathan Proulx]

We've been using Ceph as ephemeral backend (and glance store and
cinder backend) for > 2 years (maybe 3 ) and have been very happy.

cinder has been rock solid on RBD side. Early on when we had 6 osd
servers we lost one in production to a memory error.  1/6 is a large
fraction to loose but Ceph handled it as designed and there was not
noticable impact to any running instances.

The ability to start VMs from snap shot of a glance image (which is
how it's implemented if glance and nova are both cinder backed) makes
start up super fast.

Having shared storage also make live migration easy so we can do
hardware maintenence (kernel and OS upgrades) without impacting
running VMs 

As of Mitaka snapshotting running VMs is also fast, though for
earlier releases VMs we suspended while the running RBD volumes was
copied down to the the hypervisors and then restarted while the
downloaded images was re-uploaded from the hypervisor to glance and
back into Ceph.  This could mean VMs down foro 15min or more if they
had large root volumes.  As I said this got fixed in Mitaka so if
you're current this is no longer a problem.

Over all We've been very happy with our switch to Ceph and I'd
definitely recommend it.

-Jon

On Wed, Dec 07, 2016 at 05:51:01PM +0300, Vahric Muhtaryan wrote:
:Hello All, 
:
:I would like to use ephemeral disks with ceph instead of on nova compute
:node. I saw that there is an option to configure it but find many different
:bugs and reports for its not working , not stable , no success at the
:instance creation time.
:Anybody In this list use ceph as an ephemeral storage without any problem ?
:Could you pls share your experiences pls ?
:
:Regards
:VM
:
:

:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] [recognition] AUC Recognition WG is being disbanded

[Jonathan Proulx]

On Tue, Dec 06, 2016 at 06:50:18PM +, Barrett, Carol L wrote:
:Congrats Shamail – Good work by the team and it’s also good to see that WGs 
come together, complete a task and go away. ☺

I'll second that.  The AUC Recognition WG is a great model for all
future Working Groups.  Clear goal, well organized steady progress and
timely completion.

Thanks All,

-Jon


:Carol
:
:From: Shamail Tahir [mailto:itzsham...@gmail.com]
:Sent: Tuesday, December 06, 2016 10:46 AM
:To: openstack-operators ; 
user-committee 
:Subject: [User-committee] [recognition] AUC Recognition WG is being disbanded
:
:Hi everyone,
:
:The team worked diligently over the last few months to identify user 
contributions that benefit the entirety of the OpenStack community and to help 
get these contributors recognized as Active User Contributors (AUC).  The AUC 
is the constituency for the User Committee and will be the voting body for User 
Committee elections in the future.
:
:With the approval of the updated by-laws and User Committee charter earlier 
today[1], we have completed our objective of defining AUC.  The User Committee 
members can take ownership of any future adjustments that need to be made to 
the AUC criteria[2].
:
:Our wiki[3] has been updated to reflect our status and I have also submitted a 
patch to cancel our weekly team meeting[4].
:
:I would like to thank everyone on the team for staying focused on the 
objective and helping us achieve this important task!
:
:AUC Recognition WG participants:
:carolbarrett
:
:dabukalam
:
:dc_mattj
:
:emagana
:
:jproulx
:
:maishsk (co-chair)
:
:meganr
:
:mrhillsman
:
:pfreund
:
:pholland
:
:rstarmer
:
:shamail (co-chair)
:
:swdevangel
:
:
:
:
:[1] 
http://lists.openstack.org/pipermail/user-committee/2016-December/001480.html
:[2] http://www.slideshare.net/ShamailXD/openstack-auc-overview
:[3] https://wiki.openstack.org/wiki/AUCRecognition
:[4] https://review.openstack.org/#/c/407637/
:
:--
:Thanks,
:Shamail Tahir
:t: @ShamailXD
:tz: Eastern Time

:___
:User-committee mailing list
:user-commit...@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee


-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] User Committee IRC Meeting - Monday March 13th

[Jonathan Proulx]

On Mon, Mar 13, 2017 at 03:45:14PM +, randy.perry...@dell.com wrote:
:Dell - Internal Use - Confidential
:Can someone post the schedule for these meeings?


Every two weeks (on odd weeks) on Monday at 1900 UTC in #openstack-meeting

http://eavesdrop.openstack.org/#User_Committee_Meeting

schedule is also on
https://wiki.openstack.org/wiki/Governance/Foundation/UserCommittee
though a bit harder to point directly at


-Jon


:From: Edgar Magana [mailto:edgar.mag...@workday.com]
:Sent: Monday, March 13, 2017 4:24 PM
:To: user-commit...@lists.openstack.org; OpenStack Operators 
; openst...@lists.openstack.org
:Subject: [Openstack-operators] User Committee IRC Meeting - Monday March 13th
:
:Dear UC Community,
:
:As you know this week we are having the Ops Meetup in Milan. A very 
significant number of members from the community is attending the event. We are 
proposing to cancel this week UC IRC meeting. We will resume in two weeks from 
now.
:
:Enjoy the sessions in Milan.
:
:Thanks,
:
:User Committee

:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [User-committee] Non-Candidacy for UC Election August 2017

[Jonathan Proulx]

Hello All,

It has been an honor and a privelege to serve on the OpenStack User
Committee.  As the longest serving member I feel it's time to make way
for the new and as such I will not be standing for ellection in the
current cycle.

If you have an interest in representing the interest of OpenStack
users please nominate yourself!

We do need candidates to serve and nominations are now open through
August 11, 05:59 UTC

Three seats (of five) are up for election. see
https://governance.openstack.org/uc/reference/uc-election-aug2017.html

from that page:

Self-nomination is common, no third party nomination is required. They
do so by sending an email to the user-commit...@lists.openstack.org
mailing-list, with the subject: “UC candidacy” by August 11, 05:59
UTC. The email can include a description of the candidate
platform. The candidacy is then confirmed by one of the election
officials, after verification of the electorate status of the
candidate.


-Jon

Jonathan Proulx
Sr. Technical Architect
MIT CSAIL

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Experience with Cinder volumes as root disks?

[Jonathan Proulx]

Hi Conrad,

We boot to ephemeral disk by default but our ephemeral disk is Ceph
RBD just like out cinder volumes.

Using Ceph for Cinder Volumes and Glance Images storage it is possible
to very quickly create new Persistent Volumes from Glance Images
becasue on the backend it's just a CoW snapshot operation (even though
we use seperate pools for ephemeral disk, persistent volumes, and
images). This is also what happens for ephemeral boting which is much
faster than copying image to local disk on hypervisor first so we get
quick starts and relatively easy live migrations (which we use for
maintenance like hypervisor reboots and reinstalls).

I don't know how to make it the "default" but ceph definately makes
it faster. Other backends I've used basicly mount the raw storage
volume download the imaage then 'dd' in into place which is
painfully slow.

As to why ephemeral rather than volume backed by default it's much
easier to boot amny copies of the same thing and be sure they're the
same using ephemeral storage and iamges or snapshots.  Volume backed
instances tend to drift.

That said workign in a research lab many of my users go for the more
"Pet" like persistent VM workflow.  We just manage it with docs and
education, though there is always someone who misses the red flashing
"ephemeral means it gets deleted when you turn it off" sign and is
sad.

-Jon

On Tue, Aug 01, 2017 at 02:50:45PM +, Kimball, Conrad wrote:
:In our process of standing up an OpenStack internal cloud we are facing the 
question of ephemeral storage vs. Cinder volumes for instance root disks.
:
:As I look at public clouds such as AWS and Azure, the norm is to use 
persistent volumes for the root disk.  AWS started out with images booting onto 
ephemeral disk, but soon after they released Elastic Block Storage and ever 
since the clear trend has been to EBS-backed instances, and now when I look at 
their quick-start list of 33 AMIs, all of them are EBS-backed.  And I'm not 
even sure one can have anything except persistent root disks in Azure VMs.
:
:Based on this and a number of other factors I think we want our user normal / 
default behavior to boot onto Cinder-backed volumes instead of onto ephemeral 
storage.  But then I look at OpenStack and its design point appears to be 
booting images onto ephemeral storage, and while it is possible to boot an 
image onto a new volume this is clumsy (haven't found a way to make this the 
default behavior) and we are experiencing performance problems (that admittedly 
we have not yet run to ground).
:
:So ...
:
:* Are other operators routinely booting onto Cinder volumes instead of 
ephemeral storage?
:
:* What has been your experience with this; any advice?
:
:Conrad Kimball
:Associate Technical Fellow
:Chief Architect, Enterprise Cloud Services
:Application Infrastructure Services / Global IT Infrastructure / Information 
Technology & Data Analytics
:conrad.kimb...@boeing.com
:P.O. Box 3707, Mail Code 7M-TE
:Seattle, WA  98124-2207
:Bellevue 33-11 bldg, office 3A6-3.9
:Mobile:  425-591-7802
:

:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [User-committee] Non-Candidacy for UC Election August 2017

[Jonathan Proulx]

On Thu, Aug 03, 2017 at 06:45:08PM -0400, Shamail Tahir wrote:
:Jon,
:
:Thank you for giving so much time and effort to helping our user community 
grow and prosper. I hope this is a "see you soon" moment and not a "goodbye".  
It's been a pleasure and an honor being on same committee.

Don't worry you can't get rid of me that easily :)

-Jon

:Thanks again for all that you have done!
:
:Sincerely,
:Shamail 
:
:> On Aug 3, 2017, at 4:11 PM, Jonathan Proulx <j...@csail.mit.edu> wrote:
:> 
:> Hello All,
:> 
:> It has been an honor and a privelege to serve on the OpenStack User
:> Committee.  As the longest serving member I feel it's time to make way
:> for the new and as such I will not be standing for ellection in the
:> current cycle.
:> 
:> If you have an interest in representing the interest of OpenStack
:> users please nominate yourself!
:> 
:> We do need candidates to serve and nominations are now open through
:> August 11, 05:59 UTC
:> 
:> Three seats (of five) are up for election. see
:> https://governance.openstack.org/uc/reference/uc-election-aug2017.html
:> 
:> from that page:
:> 
:> Self-nomination is common, no third party nomination is required. They
:> do so by sending an email to the user-commit...@lists.openstack.org
:> mailing-list, with the subject: “UC candidacy” by August 11, 05:59
:> UTC. The email can include a description of the candidate
:> platform. The candidacy is then confirmed by one of the election
:> officials, after verification of the electorate status of the
:> candidate.
:> 
:> 
:> -Jon
:> 
:> Jonathan Proulx
:> Sr. Technical Architect
:> MIT CSAIL
:> 
:> ___
:> User-committee mailing list
:> user-commit...@lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [neutron] ML2/OVS dropping packets?

[Jonathan Proulx]

Hi All,

I have a very busy VM (well one of my users does I don't have access
but do have cooperative and copentent admin to interact with on th
eother end).

At peak times it *sometimes* misses packets.  I've been didding in for
a bit ant it looks like they get dropped in OVS land.

The VM's main function in life is to pull down webpages from other
sites and analyze as requested.  During peak times ( EU/US working
hours ) it sometimes hangs some requests and sometimes fails.

Looking at traffic the out bound SYN request from VM is always good
and returning ACK always gets to physical interface of the hypervisosr
(on a provider vlan).

When packets get dropped they do not make it to the qvo-XX on
the integration bridge.

My suspicion is that OVS isn't keeping up eth1-br flow rules remaping
from external to internal vlan-id but neither quite sure how to prove
that or what to do about it.

My initial though had been to blame contrack but drops are happening
before the iptables rules and while there's a lot of connections on
this hypervisor: 

net.netfilter.nf_conntrack_count = 351880

There should be plent of overhead to handle:

net.netfilter.nf_conntrack_max = 1048576

Anyone have thought son where to go with this?

version details:
Ubuntu 14.04
OpenStack Mitaka
ovs-vsctl (Open vSwitch) 2.5.0

Thanks,
-Jon

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [neutron] ML2/OVS dropping packets?

[Jonathan Proulx]

So this all gets more interesting the packets aren't lost they get
routed (switched?) to the wrong interface...


The VM has two interfaces on the same network. Not sure this makes
sense and wes done because this was a straight physical to virtual
migration.  But seems like it should work

so VM is sending SYN from it's (vm)eth0 -> tap0 -> qvb0 -> qvo0 -> int-eth1-br
-> phy-eht1-br -> (hypervisor)eth1 -> WORLD

but the ACK is coming back (hypervisor)eth1 ->  phy-eht1-br ->
int-eth1-br -> qvo1 !!! -> qvb1 -> tap1 where presumablely sec-group
rules see it as invalid and drop it.

This is quite odd.  Default route on VM is through eth0 where packets
are originatine and where teh ipv4 address it should return to is.

really puzzled why OVS is sending packets back through wrong path.

on the one hand I want to say stop doing that just put both addresses
on one port, on the other I see no reason why it shouldn't work.

-Jon
 

On Wed, Jun 21, 2017 at 05:35:02PM +0100, Stig Telfer wrote:
:Hi Jon -
:
:From what I understand, while you might have gone to the trouble of 
configuring a lossless data centre ethernet, that guarantee of packet loss ends 
at the hypervisor. OVS (and other virtual switches) will drop packets rather 
than exert back pressure.
:
:I saw a useful paper from IBM Zurich on developing a flow-controlled virtual 
switch:
:
:http://researcher.ibm.com/researcher/files/zurich-DCR/Got%20Loss%20Get%20zOVN.pdf
 
<http://researcher.ibm.com/researcher/files/zurich-DCR/Got%20Loss%20Get%20zOVN.pdf>
:
:It’s a bit dated (2013) but may still apply.
:
:If you figure out a way of preventing this with modern OVS, I’d be very 
interested to know.
:
:Best wishes,
:Stig
:
:
:> On 21 Jun 2017, at 16:24, Jonathan Proulx <j...@csail.mit.edu> wrote:
:> 
:> On Wed, Jun 21, 2017 at 02:39:23AM -0700, Kevin Benton wrote:
:> :Are there any events going on during these outages that would cause
:> :reprogramming by the Neutron agent? (e.g. port updates) If not, it's likely
:> :an OVS issue and you might want to cross-post to the ovs-discuss mailing
:> :list.
:> 
:> Guess I'll have to wander deeper into OVS land.
:> 
:> No agent updates and nothing in ovs logs (at INFO), flipping to Debug
:> and there's so many messages they get dropped:
:> 
:> 017-06-21T15:15:36.972Z|00794|dpif(handler12)|DBG|Dropped 35 log messages in 
last 0 seconds (most recently, 0 seconds ago) due to excessive rate
:> 
:> /me wanders over to ovs-discuss
:> 
:> Thanks,
:> -Jon
:> 
:> :Can you check the vswitch logs during the packet loss to see if there are
:> :any messages indicating a reason? If that doesn't show anything and it can
:> :be reliably reproduced, it might be worth increasing the logging for the
:> :vswitch to debug.
:> :
:> :
:> :
:> :On Tue, Jun 20, 2017 at 12:36 PM, Jonathan Proulx <j...@csail.mit.edu> 
wrote:
:> :
:> :> Hi All,
:> :>
:> :> I have a very busy VM (well one of my users does I don't have access
:> :> but do have cooperative and copentent admin to interact with on th
:> :> eother end).
:> :>
:> :> At peak times it *sometimes* misses packets.  I've been didding in for
:> :> a bit ant it looks like they get dropped in OVS land.
:> :>
:> :> The VM's main function in life is to pull down webpages from other
:> :> sites and analyze as requested.  During peak times ( EU/US working
:> :> hours ) it sometimes hangs some requests and sometimes fails.
:> :>
:> :> Looking at traffic the out bound SYN request from VM is always good
:> :> and returning ACK always gets to physical interface of the hypervisosr
:> :> (on a provider vlan).
:> :>
:> :> When packets get dropped they do not make it to the qvo-XX on
:> :> the integration bridge.
:> :>
:> :> My suspicion is that OVS isn't keeping up eth1-br flow rules remaping
:> :> from external to internal vlan-id but neither quite sure how to prove
:> :> that or what to do about it.
:> :>
:> :> My initial though had been to blame contrack but drops are happening
:> :> before the iptables rules and while there's a lot of connections on
:> :> this hypervisor:
:> :>
:> :> net.netfilter.nf_conntrack_count = 351880
:> :>
:> :> There should be plent of overhead to handle:
:> :>
:> :> net.netfilter.nf_conntrack_max = 1048576
:> :>
:> :> Anyone have thought son where to go with this?
:> :>
:> :> version details:
:> :> Ubuntu 14.04
:> :> OpenStack Mitaka
:> :> ovs-vsctl (Open vSwitch) 2.5.0
:> :>
:> :> Thanks,
:> :> -Jon
:> :>
:> :> --
:> :>
:> :> ___
:> :> OpenStack-operators mailing list
:> :> OpenStack-operato

Re: [Openstack-operators] [neutron] ML2/OVS dropping packets?

[Jonathan Proulx]

On Wed, Jun 21, 2017 at 02:39:23AM -0700, Kevin Benton wrote:
:Are there any events going on during these outages that would cause
:reprogramming by the Neutron agent? (e.g. port updates) If not, it's likely
:an OVS issue and you might want to cross-post to the ovs-discuss mailing
:list.

Guess I'll have to wander deeper into OVS land.

No agent updates and nothing in ovs logs (at INFO), flipping to Debug
and there's so many messages they get dropped:

017-06-21T15:15:36.972Z|00794|dpif(handler12)|DBG|Dropped 35 log messages in 
last 0 seconds (most recently, 0 seconds ago) due to excessive rate

/me wanders over to ovs-discuss

Thanks,
-Jon

:Can you check the vswitch logs during the packet loss to see if there are
:any messages indicating a reason? If that doesn't show anything and it can
:be reliably reproduced, it might be worth increasing the logging for the
:vswitch to debug.
:
:
:
:On Tue, Jun 20, 2017 at 12:36 PM, Jonathan Proulx <j...@csail.mit.edu> wrote:
:
:> Hi All,
:>
:> I have a very busy VM (well one of my users does I don't have access
:> but do have cooperative and copentent admin to interact with on th
:> eother end).
:>
:> At peak times it *sometimes* misses packets.  I've been didding in for
:> a bit ant it looks like they get dropped in OVS land.
:>
:> The VM's main function in life is to pull down webpages from other
:> sites and analyze as requested.  During peak times ( EU/US working
:> hours ) it sometimes hangs some requests and sometimes fails.
:>
:> Looking at traffic the out bound SYN request from VM is always good
:> and returning ACK always gets to physical interface of the hypervisosr
:> (on a provider vlan).
:>
:> When packets get dropped they do not make it to the qvo-XX on
:> the integration bridge.
:>
:> My suspicion is that OVS isn't keeping up eth1-br flow rules remaping
:> from external to internal vlan-id but neither quite sure how to prove
:> that or what to do about it.
:>
:> My initial though had been to blame contrack but drops are happening
:> before the iptables rules and while there's a lot of connections on
:> this hypervisor:
:>
:> net.netfilter.nf_conntrack_count = 351880
:>
:> There should be plent of overhead to handle:
:>
:> net.netfilter.nf_conntrack_max = 1048576
:>
:> Anyone have thought son where to go with this?
:>
:> version details:
:> Ubuntu 14.04
:> OpenStack Mitaka
:> ovs-vsctl (Open vSwitch) 2.5.0
:>
:> Thanks,
:> -Jon
:>
:> --
:>
:> ___
:> OpenStack-operators mailing list
:> OpenStack-operators@lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:>

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] It's time...

[Jonathan Proulx]

You've done amazing things for OpenStack can't wait to see what amzing
thing you do next.

If I'd known sooner would have agreed to 60hrs of cattle class travel
to see you off...unfortunately as it is I won't be in Sydney.  I do
sincerely hope our paths cross again.

All the best,
-Jon

On Wed, Oct 04, 2017 at 10:20:57PM +0800, Tom Fifield wrote:
:Hi all,
:
:Tom here, on a personal note.
:
:It's quite fitting that this November our summit is in Australia :)
:
:I'm hoping to see you there because after being part of 15 releases, and
:travelling the equivalent of a couple of round trips to the moon to witness
:OpenStack grow around the world, the timing is right for me to step down as
:your Community Manager.
:
:We've had an incredible journey together, culminating in the healthy
:community we have today. Across more than 160 countries, users and developers
:collaborate to make clouds better for the work that matters. The diversity of
:use is staggering, and the scale of resources being run is quite significant.
:We did that :)
:
:
:Behind the scenes, I've spent the past couple of months preparing to
:transition various tasks to other members of the Foundation staff. If you see
:a new name behind an openstack.org email address, please give them due
:attention and care - they're all great people. I'll be around through to year
:end to shepherd the process, so please ping me if you are worried about
:anything.
:
:Always remember, you are what makes OpenStack. OpenStack changes and thrives
:based on how you feel and what work you do. It's been a privilege to share
:the journey with you.
:
:
:
:So, my plan? After a decade of diligent effort in organisations
:euphemistically described as "minimally-staffed", I'm looking forward to
:taking a decent holiday. Though, if you have a challenge interesting enough
:to wrest someone from a tropical beach or a misty mountain top ... ;)
:
:
:There are a lot of you out there to whom I remain indebted. Stay in touch to
:make sure your owed drinks make it to you!
:
:+886 988 33 1200
:t...@tomfifield.net
:https://www.linkedin.com/in/tomfifield
:https://twitter.com/TomFifield
:
:
:Regards,
:
:
:
:Tom
:
:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Should we allow passing new user_data during rebuild?

[Jonathan Proulx]

On Tue, Oct 03, 2017 at 08:29:45PM +, Jeremy Stanley wrote:
:On 2017-10-03 16:19:27 -0400 (-0400), Jonathan Proulx wrote:
:[...]
:> This works in our OpenStack where it's our IP space so PTR record also
:> matches, not so well in public cloud where we can reserve an IP and
:> set forward DNS but not control its reverse mapping.
:[...]
:
:Not that it probably helps, but I consider any public cloud which
:doesn't give you some means of automatically setting reverse DNS
:(either through an API or delegation to your own nameservers) to be
:thoroughly broken, at least for Internet-facing use cases.

we wander off topic...and I wander waaay off topic below...

but I have exactly 1 instance in AWS where I
care about this, perhaps I just don't care enough to have found the
answer or perhaps for count of 1 it's not worth solving.


Then again perhaps AWS is just actually the trash is seem to be to
me. I've been trying to like it since before there was an OpenStack,
but the more I try the less I can stand it.  People who use AWS and
complain about OpenStack UX baffle me, there's a lot OpenStack can do
to impove UX but  it's waaay better than my AWS experices. I mean it
was fewer steps to enable ipv6 on my OpenStack provider networks than
it was to get it working in my AWS VPC and neutron isn't the poster
child for simplicity.


:-- 
:Jeremy Stanley



:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


-- 


signature.asc
Description: PGP signature
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [nova] Should we allow passing new user_data during rebuild?

[Jonathan Proulx]

On Tue, Oct 03, 2017 at 01:00:13PM -0700, Clint Byrum wrote:

:It's worth noting that AD and Kerberos were definitely not designed
:for clouds that have short lived VMs, so it does not surprise me that
:treating VMs as cattle and then putting them in AD would confuse it.

For instances we have that need Kerberos keytabs we specify the fixed
IP. This works in our OpenStack where it's our IP space so PTR record also
matches, not so well in public cloud where we can reserve an IP and
set forward DNS but not control its reverse mapping.

-Jon

:Excerpts from Tim Bell's message of 2017-10-03 18:46:31 +:
:> We use rebuild when reverting with snapshots. Keeping the same IP and 
hostname avoids some issues with Active Directory and Kerberos.
:> 
:> Tim
:> 
:> -Original Message-
:> From: Clint Byrum 
:> Date: Tuesday, 3 October 2017 at 19:17
:> To: openstack-operators 
:> Subject: Re: [Openstack-operators] [nova] Should we allow passing new
user_data during rebuild?
:> 
:> 
:> Excerpts from Matt Riedemann's message of 2017-10-03 10:53:44 -0500:
:> > We plan on deprecating personality files from the compute API in a new 
:> > microversion. The spec for that is here:
:> > 
:> > https://review.openstack.org/#/c/509013/
:> > 
:> > Today you can pass new personality files to inject during rebuild, and 
:> > at the PTG we said we'd allow passing new user_data to rebuild as a 
:> > replacement for the personality files.
:> > 
:> > However, if the only reason one would need to pass personality files 
:> > during rebuild is because we don't persist them during the initial 
:> > server create, do we really need to also allow passing user_data for 
:> > rebuild? The initial user_data is stored with the instance during 
:> > create, and re-used during rebuild, so do we need to allow updating it 
:> > during rebuild?
:> > 
:> 
:> My personal opinion is that rebuild is an anti-pattern for cloud, and
:> should be frozen and deprecated. It does nothing but complicate Nova
:> and present challenges for scaling.
:> 
:> That said, if it must stay as a feature, I don't think updating the
:> user_data should be a priority. At that point, you've basically created 
an
:> entirely new server, and you can already do that by creating an entirely
:> new server.
:> 
:> ___
:> OpenStack-operators mailing list
:> OpenStack-operators@lists.openstack.org
:> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:> 
:
:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Meetups team minutes + main topics

[Jonathan Proulx]

:On Tue, Nov 21, 2017 at 9:15 AM, Chris Morgan  wrote:

:> The big topic of debate, however, was whether subsequent meetups should be
:> co-located with OpenStack PTG. This is a question for the wider OpenStack
:> operators community.

For people who attend both I thnik this would be a big win, if they
were in the same location (city anyway) but held in series.  One
(longer) trip but no scheduling conflict.

Downside I see is that makes scheduling constraints pretty tight
either for having two sponsorslocation available in a coordinated time
and place or making a much bigger ask of a single location.

Those are my thoughts, not sure if the amount to an opinion.

-Jon



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Meetups team minutes + main topics

[Jonathan Proulx]

On Tue, Nov 21, 2017 at 10:10:00AM -0700, David Medberry wrote:
:Jon,
:
:I think the Foundation staff were very very wary of extending the PTG or
:doing dual sites simultaneously due to not saving a thing logistically.
:Yes, it would conceivably save travel for folks that need to go to two
:separate events (as would the other colo options on the table) but not
:saving a thing logistically over two separate events as we have now. A six
:or seven day sprint/thing/ptg would also mean encroaching on one or both
:weekends (above and beyond travel dates) and that may really limit venue
:choices as private parties (weddings, etc) tend to book those locales on
:weekends.

Yes, that was my main concern as well.  Though I'd not extended it to
the fact the two events wouldn't fit in a single working week.  So
sounds like the logistics are just illogistical.

-Jon

:On Tue, Nov 21, 2017 at 10:06 AM, Jonathan Proulx <j...@csail.mit.edu> wrote:
:
:> :On Tue, Nov 21, 2017 at 9:15 AM, Chris Morgan <mihali...@gmail.com>
:> wrote:
:>
:> :> The big topic of debate, however, was whether subsequent meetups should
:> be
:> :> co-located with OpenStack PTG. This is a question for the wider
:> OpenStack
:> :> operators community.
:>
:> For people who attend both I thnik this would be a big win, if they
:> were in the same location (city anyway) but held in series.  One
:> (longer) trip but no scheduling conflict.
:>
:> Downside I see is that makes scheduling constraints pretty tight
:> either for having two sponsorslocation available in a coordinated time
:> and place or making a much bigger ask of a single location.
:>
:> Those are my thoughts, not sure if the amount to an opinion.
:>
:> -Jon
:>
:>
:>

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Clear Linux guest issues (maybe UEFI boot issues?)

[Jonathan Proulx]

Hi All,

This is low priority poking but I have a sort of puzzling situation.

Wanted to play with Clear linux (which is uefi only) to see what it
was about  so grabbed some images and obviously tried to throw them on
my OpenStack cloud.

After learning how to enable EUFI booting (hw_firmware_type='uefi' in
image metadata and ovfm package on hypervisors) it still just hangs
trying to boot, no serial out, uninialized VNC console, no ping.

Running manually under KVM on my workstation (same Ubuntu 16.04 as
hypervisors) I can get it to go.

Nova generated qemu command: https://pastebin.com/VarvB6Cs
Simpler manual command (works): https://pastebin.com/BfJpXYRm

any clues?

thanks,
-Jon
-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [openstack-dev] Upstream LTS Releases

[Jonathan Proulx]

On Thu, Nov 09, 2017 at 04:34:24PM +, Jeremy Stanley wrote:
:On 2017-11-08 23:15:15 -0800 (-0800), Clint Byrum wrote:
:[...]
:> The biggest challenge will be ensuring that the skip-level upgrades
:> work. The current grenade based upgrade jobs are already quite a bear to
:> keep working IIRC. I've not seen if chef or any of the deployment projects
:> test upgrades like that.
:[...]
:
:Another challenge which has been mostly hand-waved away at this
:stage is the distro support piece. Queens is being tested on Ubuntu
:16.04 but our "S" release will likely be tested on Ubuntu 18.04
:instead... so effective testing for a skip-level upgrade between
:those two LTS releases will _also_ involve in-place upgrading of
:Ubuntu.

Having done inplace Ubuntu upgrades from 12.04->14.04->16.04
underneath "production"* openstack I'm not too worried about the
mechanics of that.

Currently for Ubuntu case when you get to a release boundary you need
to bring old Distro Release upto Newest OpenStack then move to new
Distro Release.

You can push production load to another node, reinstall new version of
Ubuntu with current configs then move load back. This does make some
assumptions about the deployed architecture but hopefully if nonstop
cloud is what you're after you've deployed something that's resilient
to single node faults which is basically what a Distro upgrade takes.

-Jon

:-- 
:Jeremy Stanley
:
:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Community Documentation - first anchor point

[Jonathan Proulx]

On May 25, 2018 5:30:40 AM PDT, Doug Hellmann  wrote:
>Excerpts from Jonathan D. Proulx's message of 2018-05-24 07:19:29
>-0700:
>> 
>> My intention based on current understandign would be to create a git
>> repo called "osops-docs" as this fits current naming an thin initial
>> document we intend to put there and the others we may adopt from
>> docs-team.
>
>Normally I would say "yay, consistency!" In this case, let's verify
>that that name isn't going to have an undesirable effect when the
>content is published.
>
>I know the default destination directory for the publish job is
>taken from the repository name, which would mean we would have a
>URL like docs.openstack.org/osops-docs. I don't know if there is a
>way to override that, but the infra team will know. So, if you want
>a URL like docs.o.o/operations-guide instead, you'll want to check
>with the infra folks before creating the repo to make sure it's set
>up in a way to get the URL you want.

Names are hard! Thanks for pointing out the implications.

-Jon

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] I/O errors on RBD after hypervisor crash.

[Jonathan Proulx]

In Proulx's Corollary to Murphy's Law, just after hitting send I tried
something that "worked".

I noticed the volume shared nothing with the image it was based on
so tried "flattening" it just to try something.

Oddly that worked, that or just having waited in power off state for
an hour wile I was at lunch.

Still have no theory on why it broke or how that could be a fix...if
anyone else does please do tell :)

Thanks,
-JOn

On Mon, Apr 30, 2018 at 12:58:16PM -0400, Jonathan Proulx wrote:
:Hi All,
:
:I have a VM with ephemeral root on RBD spewing I/O erros on boot after
:hypervisor crash.  I've (unfortunately) seen a lot of hypervisors go
:down badly with lots of VMs on them and this is a new one on me.
:
:I can 'rbd export' the volume and I get a clean filesystem.
:
:version details
:
:OpenStack: Mitaka
:Host OS:   Ubuntu 16.04
:Ceph:  Luminous (12.2.4)
:
:after booting to initrd VM shows:
:
:end_request: I/O error, dev vda, sector 
:
:Tried hard reboot, tried rescue (in which case vdb shows same
:issue) tried migrating to different hypervisor and all have consistent
:failure.
:
:I do have writeback caching enable on the crashed hypervisor so I can
:imaging filesystem corruption, but not this type of I/O error.
:
:Also if the rbd volume doesn't seem to be dammaged since I could dump
:it to an iamge and see correct partioning and filesystems.
:
:Anyone seen this before? I have the bits since the export worked but
:concerned about possibility of recurrence.
:
:Thanks,
:-Jon
:
:-- 

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] I/O errors on RBD after hypervisor crash.

[Jonathan Proulx]

Hi All,

I have a VM with ephemeral root on RBD spewing I/O erros on boot after
hypervisor crash.  I've (unfortunately) seen a lot of hypervisors go
down badly with lots of VMs on them and this is a new one on me.

I can 'rbd export' the volume and I get a clean filesystem.

version details

OpenStack: Mitaka
Host OS:   Ubuntu 16.04
Ceph:  Luminous (12.2.4)

after booting to initrd VM shows:

end_request: I/O error, dev vda, sector 

Tried hard reboot, tried rescue (in which case vdb shows same
issue) tried migrating to different hypervisor and all have consistent
failure.

I do have writeback caching enable on the crashed hypervisor so I can
imaging filesystem corruption, but not this type of I/O error.

Also if the rbd volume doesn't seem to be dammaged since I could dump
it to an iamge and see correct partioning and filesystems.

Anyone seen this before? I have the bits since the export worked but
concerned about possibility of recurrence.

Thanks,
-Jon

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Custom libvirt fragment for instance type?

[Jonathan Proulx]

On Tue, Jan 16, 2018 at 03:49:25PM -0500, Jonathan Proulx wrote:
:On Tue, Jan 16, 2018 at 08:42:00PM +, Tim Bell wrote:
::If you want to hide the VM signature, you can use the img_hide_hypervisor_id 
property 
(https://docs.openstack.org/python-glanceclient/latest/cli/property-keys.html)
:
:Thanks Tim, I believe that's the magic I was looking for.

Unfortunately settign that doesn't appear to do anything way back here
in Mitaka land :(

Oh well reason 128478 that I need to take that series of leaps I
suppose.


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Custom libvirt fragment for instance type?

[Jonathan Proulx]

On Tue, Jan 16, 2018 at 08:42:00PM +, Tim Bell wrote:
:If you want to hide the VM signature, you can use the img_hide_hypervisor_id 
property 
(https://docs.openstack.org/python-glanceclient/latest/cli/property-keys.html)

Thanks Tim, I believe that's the magic I was looking for.

-Jon

:Tim
:
:-Original Message-
:From: jon 
:Date: Tuesday, 16 January 2018 at 21:14
:To: openstack-operators 
:Subject: [Openstack-operators] Custom libvirt fragment for instance type?
:
:Hi All,
:
:Looking for a way to inject:
:
: 
:
:  
:
: 
:
:into the libvirt.xml for instances of a particular flavor.
:
:My needs could also be met by attatching it to the glance image or if
:needs be per hypervisor.
:
:My Googling is not turning up anything.  Is there any way to set
:arbitray (or this particular) Libvirt/KVM freature?
:
:Thanks,
:-Jon
:
:-- 
:
:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
:
:

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-ansible] galera issues with mitaka-eol

[Jonathan Proulx]

Hi All,

In my continuing quest to install an OSA cluster with mitaka-eol in
hopes of digging our to a non-eol release eventually I've hit another
snag...

setup-hosts plays out fine
setup-infrastructure chodes soem where in galera-install

the 1st galera container gets properly bootstrapped into a 1 node
cluster.

the 2nd container (1st that needs replication) fails with:

180802 14:59:49 [Warning] WSREP: Gap in state sequence. Need state transfer.
180802 14:59:49 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' 
--address '172.29.238.84' --datadir
 '/var/lib/mysql/'   --parent '3719' --binlog '/var/lib/mysql/mariadb-bin' '
WSREP_SST: [ERROR] FATAL: The innobackupex version is 1.5.1. Needs 
xtrabackup-2.3.5 or higher to perform SST (20180
802 14:59:50.235)
180802 14:59:50 [ERROR] WSREP: Failed to read 'ready ' from: 
wsrep_sst_xtrabackup-v2 --role 'joiner' --addres
s '172.29.238.84' --datadir '/var/lib/mysql/'   --parent '3719' --binlog 
'/var/lib/mysql/mariadb-bin' 
Read: '(null)'
180802 14:59:50 [ERROR] WSREP: Process completed with error: 
wsrep_sst_xtrabackup-v2 --role 'joiner' --address '172
.29.238.84' --datadir '/var/lib/mysql/'   --parent '3719' --binlog 
'/var/lib/mysql/mariadb-bin' : 2 (No such file o
r directory)
180802 14:59:50 [ERROR] WSREP: Failed to prepare for 'xtrabackup-v2' SST. 
Unrecoverable.
180802 14:59:50 [ERROR] Aborting

The container has:

percona-xtrabackup-22/unknown,now 2.2.13-1.trusty amd64 [installed]

which is clearly the source of sadness. Seem seems bit odd that a very
default vanilla install woudl tangle like this but it is EOL and
perhaps the world has changed around it. 

What is the most OSA friendly way digging out of this?

My current prod cluster (which I eventually want this to take over) is
using

wsrep_sst_method = mysqldump

So I could set some overrides and probably make that go in OSA land
but I'd rather converge toward "noraml OSA way" unless there's
explict local reason to change from defaults and in this case I don't
have any particular need for mysqldump over xtrabackup.

-Jon

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Community Documentation - first anchor point

[Jonathan Proulx]

Hi All...

I'm still a little confused by the state of this :)

I know I made some promises then got distracted the looks like Sean
stepped up and got things a bit further, but where is it now?  Do we
have an active repo?

It would be nice to have the repo in place before OPs meetup.

-Jon

On Tue, Jun 26, 2018 at 07:40:33PM -0500, Amy Marrich wrote:
:Sean put together some really great things here and I do think the SiG
:might be the way to go as far as ownership for the repos and the plan looks
:pretty complete. I've offered to do the Git and Gerrit Lunch and Learn at
:the OPS mmetup if needed to help get folks set up and going.
:
:Amy (spotz)
:
:On Tue, Jun 26, 2018 at 11:42 AM, Sean McGinnis 
:wrote:
:
:> Reviving this thread with a fresh start. See below for the original.
:>
:> To recap, the ops community is willing to take over some of the operator
:> documentation that is no longer available due to the loss of documentation
:> team
:> resources. From discussions, there needs to be some official governance
:> over
:> this operator owned repo (or repos) so it is recommended that a sig be
:> formed.
:> The repos can be created in the meantime, but consideration needs to be
:> taken
:> about naming as by default, the repo name is what is reflected in the
:> documentation publishing location.
:>
:> SIG Formation
:> -
:> There were a couple suggestions on naming and focus for this sig, but I
:> would
:> like to make a slightly different proposal. I would actually like to see a
:> sig-operator group formed. We have repos for operator tools and other
:> useful
:> things and we have a mix of operators, vendors, and others that work
:> together
:> on things like the ops meetup. I think it would make sense to make this
:> into an
:> official SIG that could have a broader scope than just documentation.
:>
:> Docs Repos
:> --
:> Doug made a good suggestion that we may want these things published under
:> something like docs.openstack.org/operations-guide. So based on this, I
:> think
:> for now at least we should create an opestack/operations-guide repo that
:> will
:> end up being owned by this SIG. I would expect most documentation
:> generated or
:> owned by this group would just be located somewhere under that repo, but
:> if the
:> need arises we can add additional repos.
:>
:> There are other ops repos out there right now. I would expect the
:> ownership of
:> those to move under this sig as well, but that is a seperate and less
:> pressing
:> concern at this point.
:>
:> Bug Tracking
:> 
:> There should be some way to track tasks and needs for this documentation
:> and
:> any other repos that are moved under this sig. Since it is the currently
:> planned direction for all OpenStack projects (or at least there is a vocal
:> desire for it to be) I think a Storyboard project should be created for
:> this
:> SIG's activities.
:>
:> Plan
:> 
:> So to recap above, I would propose the following actions be taken:
:>
:> 1. Create sig-operators as a group to manage operator efforts at least
:> related
:>to what needs to be done in repos.
:> 2. Create an openstack/operations-guide repo to be the new home of the
:>operations documentation.
:> 3. Create a new StoryBoard project to help track work in these repos
:> x. Document all this.
:> 9. Profit!
:>
:> I'm willing to work through the steps to get these things set up. Please
:> give
:> feedback if this proposed plan makes sense or if there is anything
:> different
:> that would be preferred.
:>
:> Thanks,
:> Sean
:>
:> On Wed, May 23, 2018 at 06:38:32PM -0700, Chris Morgan wrote:
:> > Hello Everyone,
:> >
:> > In the Ops Community documentation working session today in Vancouver, we
:> > made some really good progress (etherpad here:
:> > https://etherpad.openstack.org/p/YVR-Ops-Community-Docs but not all of
:> the
:> > good stuff is yet written down).
:> >
:> > In short, we're going to course correct on maintaining the Operators
:> Guide,
:> > the HA Guide and Architecture Guide, not edit-in-place via the wiki and
:> > instead try still maintaining them as code, but with a different, new set
:> > of owners, possibly in a new Ops-focused repo. There was a strong
:> consensus
:> > that a) code workflow >> wiki workflow and that b) openstack core docs
:> > tools are just fine.
:> >
:> > There is a lot still to be decided on how where and when, but we do have
:> an
:> > offer of a rewrite of the HA Guide, as long as the changes will be
:> allowed
:> > to actually land, so we expect to actually start showing some progress.
:> >
:> > At the end of the session, people wanted to know how to follow along as
:> > various people work out how to do this... and so for now that place is
:> this
:> > very email thread. The idea is if the code for those documents goes to
:> live
:> > in a different repo, or if new contributors turn up, or if a new version
:> we
:> > will announce/discuss it here until such time as we have a better home

Re: [Openstack-operators] Ops Community Documentation - first anchor point

[Jonathan Proulx]

On Tue, Aug 21, 2018 at 02:27:44PM -0500, Sean McGinnis wrote:
:On Tue, Aug 21, 2018 at 03:14:48PM -0400, Jonathan Proulx wrote:
:Hey Jon,
:
:Pretty much everything is in place now. There is one outstanding patch to
:officially add things under the SIG governance here:
:
:https://review.openstack.org/#/c/591248/
:
:That's a formality that needs to be done, but we do have the content being
:published to the site:
:
:https://docs.openstack.org/operations-guide/
:
:And we have the repo set up and ready for updates to be proposed:
:
:http://git.openstack.org/cgit/openstack/operations-guide
:
:Our next step is to start encouraging contributions from the community. This
:would be a great things to discuss at the PTG, and I now realize that I didn't
:add this obvious thing to our ops PTG planning etherpad. I will add it there
:and hopefully we can go over some of the content and get some more interest in
:contributing to it.


Thanks for picking up my slack there :)

-Jon



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Custom libvirt fragment for instance type?

[Jonathan Proulx]

Hi All,

Looking for a way to inject:

 

  

 

into the libvirt.xml for instances of a particular flavor.

My needs could also be met by attatching it to the glance image or if
needs be per hypervisor.

My Googling is not turning up anything.  Is there any way to set
arbitray (or this particular) Libvirt/KVM freature?

Thanks,
-Jon

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Session Proposals for Vancouver Forum

[Jonathan Proulx]

Thanks for getting this kicked off Erik.  The two things you have up
to start (fast forward upgrades, and extended maintenance) are the
exact two things I want out of my trip to YVR.  At least understanding
current 'state of the art' and helping advance that in the right
directions best I can.

Thanks,
-Jon

On Tue, Apr 10, 2018 at 11:07:34AM -0400, Erik McCormick wrote:
:Greetings Ops,
:
:We are rapidly approaching the deadline for Forum session proposals
:(This coming Sunday, 4/15) and we have been rather lax in getting the
:process started from our side. I've created an etherpad here for
:everyone to put up session ideas.
:
:https://etherpad.openstack.org/p/YYZ-forum-ops-brainstorming
:
:Given the late date, please post your session ideas ASAP, and +1 those
:that you have interest in. Also if you are willing to moderate the
:session, put your name on it as you'll see on the examples already
:there.
:
:Moderating is easy, gets you a pretty little speaker sticker on your
:badge, and lets you go get your badge at the Speaker pickup line. It's
:also a good way to get AUC status and get more involved in the
:community. It's fairly painless, and only a few of us bite :).
:
:I'd like to wrap this up by Friday so we can weed out duplicates from
:others proposals and get them into the topic submission system with a
:little time to spare. It would be helpful to have moderators submit
:their own so everything credits properly. The submission system is at
:http://forumtopics.openstack.org/. If you don't already have an
:account and would like to moderate, go set one up.
:
:I'm looking forward to seeing lots of you in Vancouver!
:
:Cheers,
:Erik
:
:___
:OpenStack-operators mailing list
:OpenStack-operators@lists.openstack.org
:http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

-- 

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Session Proposals for Vancouver Forum

[Jonathan Proulx]

On Tue, Apr 10, 2018 at 11:31:55AM -0400, Erik McCormick wrote:
:On Tue, Apr 10, 2018 at 11:19 AM, Jonathan Proulx <j...@csail.mit.edu> wrote:
:>
:> Thanks for getting this kicked off Erik.  The two things you have up
:> to start (fast forward upgrades, and extended maintenance) are the
:> exact two things I want out of my trip to YVR.  At least understanding
:> current 'state of the art' and helping advance that in the right
:> directions best I can.
:>
:
:I see Tony Breeds has posted the Extended Maintenance session already
:which is awesome. I'm actually considering a Part I and Part II for
:FFU as we had a packed house and not nearly enough time in SYD. I'm
:just not sure how to structure it or break it down.

Off the top of my head perhaps a current state session focusing on
what can be done with existing releases and a forward looking session
on how to improve that experience?

My $0.02,
-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack "S" Release Naming Preliminary Results

[Jonathan Proulx]

On Wed, Mar 21, 2018 at 08:32:38PM -0400, Paul Belanger wrote:

:6. Spandau  loses to Solar by 195–88, loses to Springer by 125–118

Given this is at #6 and formal vetting is yet to come it's probably
not much of an issue, but "Spandau's" first association for many will
be Nazi war criminals via Spandau Prison
https://en.wikipedia.org/wiki/Spandau_Prison

So best avoided to say the least.

-Jon

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Ops Meetup, Co-Location options, and User Feedback

[Jonathan Proulx]

On Thu, Mar 22, 2018 at 09:02:48PM -0700, Yih Leong, Sun. wrote:
:I support the ideas to try colocating the next Ops Midcycle and PTG.
:Although scheduling could be a potential challenge but it worth give it a
:try.
:
:Also having an joint social event in the evening can also help Dev/Ops to
:meet and offline discussion. :)

Agreeing stongly with Matt and Melvin's comments about Forum -vs-
PTG/OpsMidcycle

PTG/OpsMidcycle (as I see them) are about focusing inside teams to get
work done ("how" is a a good one word I think). The advantage of
colocation is for cross team questions like "we're thinking of doing
this thing this way, does this have any impacts on your work my might
not have considered", can get a quick respose in the hall, at lunch,
or over beers as Yih Leong suggests.

Forum has become about coming to gather across groups for more
conceptual  "what" discussions.

So I also thing they are very distinct and I do see potential benefits
to colocation.

We do need to watch out for downsides.  The concerns around colocation
seemed mostly about larger events costing more and being generally
harder to organize.  If we try we will find out if there is merit to
this concern, but (IMO) it is  important to keep both of the
events as cheap and simple as possible.

-Jon

:
:On Thursday, March 22, 2018, Melvin Hillsman  wrote:
:
:> Thierry and Matt both hit the nail on the head in terms of the very
:> base/purpose/point of the Forum, PTG, and Ops Midcycles and here is my +2
:> since I have spoke with both and others outside of this thread and agree
:> with them here as I have in individual discussions.
:>
:> If nothing else I agree with Jimmy's original statement of at least giving
:> this a try.
:>
:> On Thu, Mar 22, 2018 at 4:54 PM, Matt Van Winkle 
:> wrote:
:>
:>> Hey folks,
:>> Great discussion!  There are number of points to comment on going back
:>> through the last few emails.  I'll try to do so in line with Theirry's
:>> latest below.  From a User Committee perspective (and as a member of the
:>> Ops Meetup planning team), I am a convert to the idea of co-location, but
:>> have come to see a lot of value in it.  I'll point some of that out as I
:>> respond to specific comments, but first a couple of overarching points.
:>>
:>> In the current model, the Forum sessions are very much about WHAT the
:>> software should do.  Keeping the discussions focused on behavior, feature
:>> and function has made it much easier for an operator to participate
:>> effectively in the conversation versus the older, design sessions, that
:>> focused largely on blueprints, coding approaches, etc.  These are HOW the
:>> developers should make things work and, now, are a large part of the focus
:>> of the PTG.  I realize it's not that cut and dry, but current model has
:>> allowed for this division of "what" and "how" in many areas, and I know
:>> several who have found it valuable.
:>>
:>> The other contextual thing to remember is the PTG was the effective
:>> combining of all the various team mid-cycle meetups that were occurring.
:>> The current Ops mid-cycle was born in that same period.  While it's purpose
:>> was a little different, it's spirit is the same - gather a team (in this
:>> case operators) together outside the hustle and bustle of a summit to
:>> discuss common issues, topics, etc.  I'll also point out, that they have
:>> been good vehicles in the Ops community to get new folks integrated.  For
:>> the purpose of this discussion, though, one could argue this is just
:>> bringing the last mid-cycle event in to the fold.
:>>
:>> On 3/21/18, 4:40 AM, "Thierry Carrez"  wrote:
:>>
:>> Doug Hellmann wrote:
:>> > Excerpts from Tim Bell's message of 2018-03-20 19:48:31 +:
:>> >>
:>> >> Would we still need the same style of summit forum if we have the
:>> >> OpenStack Community Working Gathering? One thing I have found with
:>> >> the forum running all week throughout the summit is that it tends
:>> >> to draw audience away from other talks so maybe we could reduce the
:>> >> forum to only a subset of the summit time?
:>> >
:>> > I support the idea of having all contributors attend the contributor
:>> > event (and rebranding it to reflect that change in emphasis), but
:>> > it's not quite clear how the result would be different from the
:>> > Forum. Is it just the scheduling? (Having input earlier in the cycle
:>> > would be convenient, for sure.)
:>> >
:>> > Thierry's comment about "work sessions" earlier in the thread seems
:>> > key.
:>>
:>> Right, I think the key difference between the PTG and Forum is that
:>> one
:>> is a work event for engaged contributors that are part of a group
:>> spending time on making OpenStack better, while the other is a venue
:>> for
:>> engaging with everyone in our community.
:>>
:>> The PTG format is really organized