[Openstack-operators] Forum Brainstorming???

[Shamail Tahir]

Hi everyone,

Welcome to the topic selection process for our first Forum in Boston. If
you've participated in an ops meetup before, this should seem pretty
comfortable. If not, note that this is *not* a classic conference track
with speakers and presentations. OpenStack community members (participants
in development teams, working groups, and other interested individuals)
discuss the topics they want to cover and get alignment on and we welcome
your participation.

The Forum is for the entire community to come together; create a neutral
space rather than having separate “ops” and “dev” days. Boston marks the
start of the Queen's release cycle, where ideas and requirements will be
gathered. Users should aim to come armed with feedback from February's
Ocata release if at all possible. We aim to ensure the broadest coverage of
topics that will allow for multiple parts of the community getting together
to discuss key areas within our community/projects.

Examples of the types of discussions and some sessions that might fit
within each one:

   - *Strategic, whole-of-community discussions*, to think about the big
   picture, including beyond just one release cycle and new technologies


   - eg *Making OpenStack One Platform for containers/VMs/Bare Metal*
(Strategic
   session) the entire community congregates to share opinions on how to make
   OpenStack achieve its integration engine goal


   - *Cross-project sessions*, in a similar vein to what has happened at
   past design summits, but with increased emphasis on issues that are
   relevant to all areas of the community


   - eg *Rolling Upgrades at Scale* (Cross-Project session) – the Large
   Deployments Team collaborates with Nova, Cinder and Keystone to tackle
   issues that come up with rolling upgrades when there’s a large number of
   machines.


   - *Project-specific sessions*, where developers can ask users specific
   questions about their experience, users can provide feedback from the last
   release and cross-community collaboration on the priorities, and ‘blue sky’
   ideas for the next release.


   - *eg Neutron Pain* *Points* (Project-Specific session) – Co-organized
   by neutron developers and users. Neutron developers bring some specific
   questions they want answered, Neutron users bring feedback from the latest
   release and ideas about the future.


There are two stages to the brainstorming:
1. Starting today, set up an etherpad with your group/team, or use one
on the list and start discussing ideas you'd like to talk about at the
Forum. Then, through +1s on etherpads and mailing list discussion, work out
which ones are the most needed - just like you did prior to the ops
events.
2. Then, in a couple of weeks, we will open up a more formal web-based
tool for submission of abstracts that came out of the brainstorming on top.

We expect working groups may make their own etherpads, however the User
Committee offers a catch-all to get the widest feedback possible:
https://etherpad.openstack.org/p/BOS-UC-brainstorming

Feel free to use that, or make one for your group and add it to the list
at: https://wiki.openstack.org/wiki/Forum/Boston2017

Thanks,
User Committee 
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [gnocchi] Gnocchi, keystone, and Openstack Mitaka

[gordon chung]

can you add what version of gnocchi, gnocchiclient and oslo.policy you 
have? might be easier if open a bug[1]. i don't see anything wrong at 
first glance and i don't recall there being a similar issue in past.

[1] https://bugs.launchpad.net/gnocchi

On 23/02/17 11:54 AM, Tracy Comstock Roesler wrote:
> I’ve run into a problem with the gnocchi CLI.  Whenever I run ‘gnocchi
> status’ I get a 403 Forbidden, but I can run other commands like
> 'gnocchi resource create’ no problem.
>
> I’ve checked the policy.json and it looks like “admin” has rights to get
> status, the same as create resources.  I cannot figure out why get
> status would show a 403 forbidden, but I can run other commands just fine.
>
> [root ~] # gnocchi status --debug
> REQ: curl -g -i -X GET http://keystone:35357/v3 -H "Accept:
> application/json" -H "User-Agent: keystoneauth1/2.4.1
> python-requests/2.10.0 CPython/2.7.5"
> Starting new HTTP connection (1): keystone
> "GET /v3 HTTP/1.1" 200 277
> RESP: [200] Content-Type: application/json Content-Length: 277
> Connection: keep-alive Date: Thu, 23 Feb 2017 16:52:40 GMT Server:
> Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 Vary: X-Auth-Token
> x-openstack-request-id: req-189a8db8-6210-4735-bc66-b2dc90b00a38
> RESP BODY: {"version": {"status": "stable", "updated":
> "2016-04-04T00:00:00Z", "media-types": [{"base": "application/json",
> "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.6",
> "links": [{"href": "http://keystone:35357/v3/";, "rel": "self"}]}}
>
> Making authentication request to http://keystone:35357/v3/auth/tokens
> "POST /v3/auth/tokens HTTP/1.1" 201 3874
> REQ: curl -g -i -X GET http://gnocchi:8041/v1/status -H "User-Agent:
> keystoneauth1/2.4.1 python-requests/2.10.0 CPython/2.7.5" -H "Accept:
> application/json, */*" -H "X-Auth-Token: {SHA1}AAA"
> Starting new HTTP connection (1): gnocchi
> "GET /v1/status HTTP/1.1" 403 54
> RESP: [403] Content-Type: application/json; charset=UTF-8
> Content-Length: 54 Connection: keep-alive Server: Werkzeug/0.9.1
> Python/2.7.5 Date: Thu, 23 Feb 2017 16:52:40 GMT
> RESP BODY: {"code": 403, "description": "", "title": "Forbidden"}
>
> Forbidden (HTTP 403)
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/site-packages/cliff/app.py", line 346, in
> run_subcommand
> result = cmd.run(parsed_args)
>   File "/usr/lib/python2.7/site-packages/cliff/display.py", line 79, in run
> column_names, data = self.take_action(parsed_args)
>   File
> "/usr/lib/python2.7/site-packages/gnocchiclient/v1/status_cli.py", line
> 21, in take_action
> status = self.app.client.status.get()
>   File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/status.py",
> line 21, in get
> return self._get(self.url).json()
>   File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/base.py", line
> 37, in _get
> return self.client.api.get(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line
> 173, in get
> return self.request(url, 'GET', **kwargs)
>   File "/usr/lib/python2.7/site-packages/gnocchiclient/client.py", line
> 38, in request
> raise exceptions.from_response(resp, method)
> Forbidden: Forbidden (HTTP 403)
> Traceback (most recent call last):
>   File "/usr/bin/gnocchi", line 10, in 
> sys.exit(main())
>   File "/usr/lib/python2.7/site-packages/gnocchiclient/shell.py", line
> 211, in main
> return GnocchiShell().run(args)
>   File "/usr/lib/python2.7/site-packages/cliff/app.py", line 226, in run
> result = self.run_subcommand(remainder)
>   File "/usr/lib/python2.7/site-packages/cliff/app.py", line 346, in
> run_subcommand
> result = cmd.run(parsed_args)
>   File "/usr/lib/python2.7/site-packages/cliff/display.py", line 79, in run
> column_names, data = self.take_action(parsed_args)
>   File
> "/usr/lib/python2.7/site-packages/gnocchiclient/v1/status_cli.py", line
> 21, in take_action
> status = self.app.client.status.get()
>   File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/status.py",
> line 21, in get
> return self._get(self.url).json()
>   File "/usr/lib/python2.7/site-packages/gnocchiclient/v1/base.py", line
> 37, in _get
> return self.client.api.get(*args, **kwargs)
>   File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line
> 173, in get
> return self.request(url, 'GET', **kwargs)
>   File "/usr/lib/python2.7/site-packages/gnocchiclient/client.py", line
> 38, in request
> raise exceptions.from_response(resp, method)
> gnocchiclient.exceptions.Forbidden: Forbidden (HTTP 403)
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>

-- 
gord

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Large deployments] Neutron issues in Openstack Large deployment using DVR

[Satyanarayana Patibandla]

Hi Kevin,

After increasing the parameter values mentioned in the below mail, we are
able to create few hundreds of VMs properly. There were no errors related
to neutron. Our environment contain multiple regions. One of our team
member by mistake ran all openstack service tempest tests against the site.
After running the tempest tests, again we observed the "504 gateway
timeout" error. This time even after restarting all neutron agents related
containers the neutron CLI was not responsive. We are getting the same
gateway timeout error even after restarting all the neutron agent
containers.

We did SHOW PROCESSLIST in MySQL. we can see a lock on the agent table
query.

In the logs we can see below error.

2017-02-27 14:50:29.085 38 ERROR oslo_messaging.rpc.server DBDeadlock:
(pymysql.err.InternalError) (1205, u'Lock wait timeout exceeded; try
restarting transaction') [SQL: u'UPDATE agents SET
heartbeat_timestamp=%(heartbeat_timestamp)s WHERE agents.id =
%(agents_id)s'] [parameters: {'heartbeat_timestamp':
datetime.datetime(2017, 2, 27, 14, 46, 35, 229400), 'agents_id':
u'94535d12-4b04-42c2-8a74-f2358db41634'}]

We are using stable/ocata code in our enviornment. We had to reimage and
redeploy all the nodes to continue our testing. Could you please let us
know your thoughts on the above issue.

Thanks,
Satya.P

On Mon, Feb 27, 2017 at 12:32 PM, Satyanarayana Patibandla <
satya.patiban...@gmail.com> wrote:

> Hi,
>
> We increased api_workers,rpc_workers and metadata_workers based on the
> number of cores we are running on controller node ( the workers are half of
> the number of cores. i.e if we have 24 cores then we are running 12 workers
> for each). Increased rpc_connect_timeout to 180 and rpc_response_timeout
> to 600. As of now it seems these are fine.
>
> Let me know if you have any comments or suggestions about increasing those
> parameter values.
>
> Thanks,
> Satya.P
>
> On Mon, Feb 27, 2017 at 11:16 AM, Kevin Benton  wrote:
>
>> Thanks for following up. Would you mind sharing the parameters you had to
>> tune (db pool limits, etc) just in case someone comes across this same
>> thread in a google search?
>>
>> Thanks,
>> Kevin Benton
>>
>> On Sun, Feb 26, 2017 at 8:48 PM, Satyanarayana Patibandla <
>> satya.patiban...@gmail.com> wrote:
>>
>>> Hi Saverio,
>>>
>>> The issue seems to be related to neutron tuning. We observed the same
>>> issue with stable/ocata branch code. When we tuned few neutron parameters
>>> it is working fine.
>>> Thanks for your suggestion.
>>>
>>> Thanks,
>>> Satya.P
>>>
>>> On Wed, Feb 22, 2017 at 10:10 AM, Satyanarayana Patibandla <
>>> satya.patiban...@gmail.com> wrote:
>>>
 Hi Saverio,

 Thanks for your inputs. Will test with statable/ocata branch code and
 will share the result.

 Thanks,
 Satya.P

 On Wed, Feb 22, 2017 at 1:54 AM, Saverio Proto 
 wrote:

> Hello,
>
> I would use at least the stable/ocata branch. If you just use master
> that is not supposed to be stable, and also I am not sure if you can
> fill a bug against a specific commit in master.
>
> Saverio
>
> 2017-02-21 21:12 GMT+01:00 Satyanarayana Patibandla
> :
> > Hi Saverio,
> >
> > We have tried to create 20 VMs each time using heat template. There
> is 1 sec
> > time gap between each VM creation request. When we reached 114 VMs
> we got
> > the error mentioned in the below mail.Heat template will boot
> instance from
> > volume and it assigns floating IP to the instance.
> >
> > Except neutron-server container we restarted all the neutron agent
> > containers which are present on all network and compute nodes. We
> are using
> > kolla to deploy openstack services.
> >
> > We are using 1 month old master branch openstack code to deploy our
> > services.
> >
> > Please find the error logs in the below link.
> > http://paste.openstack.org/show/599892/
> >
> > Thanks,
> > Satya.P
> >
> > On Wed, Feb 22, 2017 at 12:21 AM, Saverio Proto 
> wrote:
> >>
> >> Hello Satya,
> >>
> >> I would fill a bug on launchpad for this issue.
> >> 114 VMs is not much. Can you identify how to trigger the issue to
> >> reproduce it ? or it just happens randomly ?
> >>
> >> When you say rebooting the network node, do you mean the server
> >> running the neutron-server process ?
> >>
> >> what version and distribution of openstack are you using ?
> >>
> >> thank you
> >>
> >> Saverio
> >>
> >>
> >> 2017-02-21 13:54 GMT+01:00 Satyanarayana Patibandla
> >> :
> >> > Hi All,
> >> >
> >> > We are trying to deploy Openstack in our production environment.
> For
> >> > networking we are using DVR with out L3 HA. We are able to create
> 114
> >> > VMs
> >> > with out any issue. After creating 114 VMs we are getting the
> below
> >> > error.
>>

[Openstack-operators] [neutron] Does your plugin support updating network provider extended attributes?

[Boden Russell]

FYI: If your neutron networking plugin supports updating network
provider extended attributes [1], please see [2].

Thanks


[1]
https://developer.openstack.org/api-ref/networking/v2/#provider-extended-attributes
[2]
http://lists.openstack.org/pipermail/openstack-dev/2017-February/112979.html

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] User Committee IRC Meeting - Monday Feb 13th

[Edgar Magana]

Community,

We have few updates for today’s agenda.

Agenda:
- Welcome UC elected members: Shamail and Melvin
- UC Chair selection
- Product WG proposal [1]
- Review previous action items
- Board + TC + UC meeting in Boston (March 8-9, 2017) – Email draft from Shamail
- Open agenda

Thanks,
Edgar

[1] 
https://docs.google.com/presentation/d/1jTlO4UdNjc5cOKboeCNhLDEdttazWWlDgNxxulUrPNg/edit


From: Edgar Magana 
Date: Thursday, February 23, 2017 at 9:15 AM
To: "user-commit...@lists.openstack.org" , 
OpenStack Operators , 
"openst...@lists.openstack.org" 
Cc: Jon Proulx , Shilla Saebi , 
Melvin Hillsman , Shamail 
Subject: Re: User Committee IRC Meeting - Monday Feb 13th

Dear UC Community,

Our next UC IRC meeting is scheduled for this Monday Feb 27th at 1900 UTC in 
(freenode) #openstack-meeting.

Agenda:
- Welcome UC elected members: Shamail and Melvin
- Review previous action items
- Board + TC + UC meeting in Boston (March 8-9, 2017)
- Open agenda

Thanks,

Edgar
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [openstack-operators] [osops] Meeting

[Melvin Hillsman]

Hey everyone,

Just a friendly reminder about OSOps meeting on today at 1400UTC in
#openstack-operators

Please be sure to review the etherpad - https://etherpad.openstack.org/p/
osops-meeting - and add agenda items or comment on existing ones.

-- 
Kind regards,

Melvin Hillsman
Ops Technical Lead
OpenStack Innovation Center

mrhills...@gmail.com
phone: (210) 312-1267
mobile: (210) 413-1659
http://osic.org

Learner | Ideation | Belief | Responsibility | Command
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators