[openstack-dev] [nova] wrong disk usage for volume-backed instances

[Tobias Urdin]

Hello,

This issue has been active since mid 2014 and still hasn't got any
resolution or fixes merged.
I'm pushing for a fix to this issue and anybody with help or feedback is
welcome.

Please see my last comment on
https://bugs.launchpad.net/nova/+bug/1469179 regarding the old review
which could be revived [1].
We have been running this patch in out testing and staging environment
but I'm still reviewing it.

We are now several people involved in this but I would like to have a
lot of opinions about reviving this review.

Best regards

[1] https://review.openstack.org/#/c/200870/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Openstack-dev] scaling nova kvm and neutron l3-ha and ml2+openvswitch

[Tobias Urdin]

Hello,

(I didn't have any luck with this question on the openstack-operators
list so I'm throwing it out here to see if I can catch anyone, according
to Assaf he knew a couple of you devs running a similar environment, if
not feel free to do what you please with this email hehe)

I'm gonna give it a try here and see if anybody has a similar setup that
could answer some questions about scaling.
We are running Liberty with Nova with KVM and Neutron L3 HA and
ML2+Openvswitch.

* How many nova instances do you have?
* How many nova compute nodes do you have?
* How many neutron nodes do you have? (Network nodes that is hosting l3
agents, dhcp agents, openvswitch-plugin etc)
* What is your overall thought on the management ability on Openvswitch?
* What issue have you had related to scaling, performance etc?

Thankful for any data, I'm trying to give my employer real world usage
information on a similar setup.
Feel free to answer me privately if you prefer but I'm sure more people
here are curious if you want to share :)

Best regards
Tobias


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Nominating mkarpin for core for the Puppet OpenStack modules

[Tobias Urdin]

Even though I dont have a vote I would like to extend a lot of gratitude
to Mykyta Karpin for
always helping out!

Congratulations and well deserved!


On 01/19/2017 11:28 PM, Alex Schultz wrote:
> Hey Puppet Cores,
>
> I would like to nominate Mykyta Karpin as a Core reviewer for the
> Puppet OpenStack modules.  He has been providing quality patches and
> reviews for some time now and I believe he would be a good addition to
> the team.  His stats for the last 90 days can be viewed here[0]
>
> Please response with your +1 or any objections. If there are no
> objections by Jan 26, I will add him to the core list.
>
> Thanks,
> -Alex
>
> [0] http://stackalytics.com/report/contribution/puppet%20openstack-group/90
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] The end of OpenStack packages in Debian?

[Tobias Urdin]

On 02/15/2017 01:46 PM, Thomas Goirand wrote:
> Hi there,
>
> It's been a while since I planed on writing this message. I couldn't
> write it because the situation makes me really sad. At this point, it
> starts to be urgent to post it.
>
> As for many other folks, Mirantis decided to end its contract with me.
> This happened when I was the most successful doing the job, with all of
> the packaging CI moved to OpenStack infra at the end of the OpenStack
> Newton cycle, after we were able to release Newton this way. I was
> hoping to start packaging on every commit for Ocata. That's yet another
> reason for me to be very frustrated about all of this. Such is life...
>
> Over the last few months, I hoped for having enough strengths to
> continue my packaging work anyway, and get Ocata packages done. But
> that's not what happened. The biggest reason for this is that I know
> that this needs to be a full time job. And at this point, I still don't
> know what my professional future will be. A company, in Barcelona, told
> me I'd get hired to continue my past work of packaging OpenStack in
> Debian, but so far, I'm still waiting for a definitive answer, so I'm
> looking into some other opportunities.
>
> All this to say that, unless someone wants to hire me for it (which
> would be the best outcome, but I fear this wont happen), or if someone
> steps in (this seems unlikely at this point), both the packaging-deb and
> the faith of OpenStack packages in Debian are currently compromised.
>
> I will continue to maintain OpenStack Newton during the lifetime of
> Debian Stretch though, but I don't plan on doing anything more. This
> means that maybe, Newton will be the last release of OpenStack in
> Debian. If things continue this way, I probably will ask for the removal
> of all OpenStack packages from Debian Sid after Stretch gets released
> (unless I know that someone will do the work).
>
> As a consequence, the following projects wont get packages even in
> Ubuntu (as they were "community maintained", which means done by me and
> later sync into Ubuntu...):
>
> - congress
> - gnocchi
> - magnum
> - mistral
> - murano
> - sahara
> - senlin
> - watcher
> - zaqar
>
> Hopefully, Canonical will continue to maintain the other 15 (more
> core...) projects in UCA.
>
> Thanks for the fish,
>
> Thomas Goirand (zigo)
>
> P,S: To the infra folks: please keep the packaging CI as it is, as it
> will be useful for the lifetime of Stretch.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
Somebody please hire Thomas, he's amazing!

Thank you for everything!


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Tobias Urdin]

Trying to trace this, tempest calls the POST /servers//action API 
endpoint for the nova compute api.

https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/floating_ips_client.py#L82

Nova then takes the requests and tries to do this floating ip association using 
the neutron server api.

http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/nova/nova-api.txt.gz

2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
[req-7f810cc7-a498-4bf4-b27e-8fc80d652785 42526a28b1a14c629b83908b2d75c647 
2493426e6a3c4253a60c0b7eb35cfe19 - default default] Unable to associate 
floating IP 172.24.5.17 to fixed IP 10.100.0.8 for instance 
d265626a-77c1-4d2f-8260-46abe548293e. Error: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out: ConnectTimeout: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out

Checking that timestamp in the neutron-server logs:
http://paste.openstack.org/show/626240/

We can see that during this timestamp right before at 23:12:30.377 and then 
after 23:12:35.611 everything seems to be doing fine.
So there is some connectivity issues to the neutron API from where the Nova API 
is running causing a timeout.

Now some more questions would be:

* Why is the return code 400? Are we being fooled or is it actually a 
connection timeout.
* Is the Neutron API stuck causing the failed connection? All talk are done 
over loopback so chance of a problem there is very low.
* Any firewall catching this? Not likely since the agent processes requests 
right before and after.

I can't find anything interesting in the overall other system logs that could 
explain that.
Back to the logs!

Best regards
Tobias


On 11/14/2017 08:35 AM, Tobias Urdin wrote:

Hello,

Same here, I will continue looking at this aswell.

Would be great if we could get some input from a neutron dev with good insight 
into the project.


Can we backtrace the timed out message from where it's thrown/returned.

Error: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out', u'code': 400}

I'm interested why we would get 400 code back, the floating ip operations 
should be async right so this would be the response from the API layer with 
information from the database that should

return more information.


Best regards

On 11/14/2017 07:41 AM, Arnaud MORIN wrote:
Hey,
Do you know if the bug appears on a specific Ubuntu / openstack version?
As far as I remember it was not related to the puppet branch?  I mean the bug 
is existing on master but also on newton puppet branches, right?

We are using Ubuntu in my company so we would love to see that continue ;)
I'll try to take a look again.

Cheers.

Le 14 nov. 2017 00:00, "Mohammed Naser" 
<mna...@vexxhost.com<mailto:mna...@vexxhost.com>> a écrit :
Hi,

Hope that everyone had safe travels and enjoyed their time at Sydney
(and those who weren't there enjoyed a bit of quiet time!).  I'm just
sending this email if anyone had a chance to look more into this (or
perhaps we can get some help if there are any Canonical folks on the
list?)

I would be really sad if we had to drop Ubuntu/Debian support because
we cannot test it.  I think there are a lot of users out there using
it!  I'd be more than happy to provide any assistance/information in
troubleshooting this.

Thank you,
Mohammed

On Thu, Nov 2, 2017 at 1:10 PM, Mohammed Naser 
<mna...@vexxhost.com<mailto:mna...@vexxhost.com>> wrote:
> On Thu, Nov 2, 2017 at 1:02 PM, Tobias Urdin 
> <tobias.ur...@crystone.com<mailto:tobias.ur...@crystone.com>> wrote:
>> I've been staring at this for almost an hour now going through all the logs
>> and I can't really pin a point from
>>
>> where that error message is generated. Cannot find any references for the
>> timed out message that the API returns or the unable to associate part.
>>
>>
>> What I'm currently staring at is why would the instance fixed ip 172.24.5.17
>> be references as a network:router_gateway port in the OVS agent logs.
>>
>> 2017-10-29 23:19:27.591 11856 INFO
>> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
>> [req-7274c6f7-18ef-420d-ad5a-9d0fe4eb35c6 - - - - -] Port
>> 053a625c-4227-41fb-9a26-45eda7bd2055 updated. Details: {'profile': {},
>> 'network_qos_policy_id': None, 'qos_policy_id': None,
>> 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id':
>> 'f9647756-41ad-4ec5-af49-daefe410815e', 'segmentation_id': None,
>> 'fixed_ips': [{'subnet_id': 'a31c7115-1f3e-4220-8bdb-981b6df2e18c',
>> 'ip_address': '172.24.5.17'}], 'device_owner': u'network:router_gateway',
>> 'physical_network': u'external', 'mac_address': 'fa:16:3e:3b:e

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Tobias Urdin]

Hello,

Same here, I will continue looking at this aswell.

Would be great if we could get some input from a neutron dev with good insight 
into the project.


Can we backtrace the timed out message from where it's thrown/returned.

Error: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out', u'code': 400}

I'm interested why we would get 400 code back, the floating ip operations 
should be async right so this would be the response from the API layer with 
information from the database that should

return more information.


Best regards

On 11/14/2017 07:41 AM, Arnaud MORIN wrote:
Hey,
Do you know if the bug appears on a specific Ubuntu / openstack version?
As far as I remember it was not related to the puppet branch?  I mean the bug 
is existing on master but also on newton puppet branches, right?

We are using Ubuntu in my company so we would love to see that continue ;)
I'll try to take a look again.

Cheers.

Le 14 nov. 2017 00:00, "Mohammed Naser" 
<mna...@vexxhost.com<mailto:mna...@vexxhost.com>> a écrit :
Hi,

Hope that everyone had safe travels and enjoyed their time at Sydney
(and those who weren't there enjoyed a bit of quiet time!).  I'm just
sending this email if anyone had a chance to look more into this (or
perhaps we can get some help if there are any Canonical folks on the
list?)

I would be really sad if we had to drop Ubuntu/Debian support because
we cannot test it.  I think there are a lot of users out there using
it!  I'd be more than happy to provide any assistance/information in
troubleshooting this.

Thank you,
Mohammed

On Thu, Nov 2, 2017 at 1:10 PM, Mohammed Naser 
<mna...@vexxhost.com<mailto:mna...@vexxhost.com>> wrote:
> On Thu, Nov 2, 2017 at 1:02 PM, Tobias Urdin 
> <tobias.ur...@crystone.com<mailto:tobias.ur...@crystone.com>> wrote:
>> I've been staring at this for almost an hour now going through all the logs
>> and I can't really pin a point from
>>
>> where that error message is generated. Cannot find any references for the
>> timed out message that the API returns or the unable to associate part.
>>
>>
>> What I'm currently staring at is why would the instance fixed ip 172.24.5.17
>> be references as a network:router_gateway port in the OVS agent logs.
>>
>> 2017-10-29 23:19:27.591 11856 INFO
>> neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
>> [req-7274c6f7-18ef-420d-ad5a-9d0fe4eb35c6 - - - - -] Port
>> 053a625c-4227-41fb-9a26-45eda7bd2055 updated. Details: {'profile': {},
>> 'network_qos_policy_id': None, 'qos_policy_id': None,
>> 'allowed_address_pairs': [], 'admin_state_up': True, 'network_id':
>> 'f9647756-41ad-4ec5-af49-daefe410815e', 'segmentation_id': None,
>> 'fixed_ips': [{'subnet_id': 'a31c7115-1f3e-4220-8bdb-981b6df2e18c',
>> 'ip_address': '172.24.5.17'}], 'device_owner': u'network:router_gateway',
>> 'physical_network': u'external', 'mac_address': 'fa:16:3e:3b:ec:c3',
>> 'device': u'053a625c-4227-41fb-9a26-45eda7bd2055', 'port_security_enabled':
>> False, 'port_id': '053a625c-4227-41fb-9a26-45eda7bd2055', 'network_type':
>> u'flat', 'security_groups': []}
>>
>>
>> Anybody else seen anything interesting?
>
> Hi Tobias,
>
> Thanks for looking out.  I've spent a lot of time and I haven't
> successfully identified much, I can add the following
>
> - This issue is intermittent in CI
> - It does *not* happen on any specific providers, I've seen it fail on
> both OVH and Rackspace.
> - Not *all* floating iP assignments fail, if you look at the logs, you
> can see it attach a few successfully before failing
>
> But yeah, I'm still quite at a loss and not having this coverage isn't fun.
>
>>
>>
>> On 10/30/2017 11:08 PM, Brian Haley wrote:
>>
>> On 10/30/2017 05:46 PM, Matthew Treinish wrote:
>>
>>  From a quick glance at the logs my guess is that the issue is related
>> to this stack trace in the l3 agent logs:
>>
>> http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/neutron/neutron-l3-agent.txt.gz?level=TRACE#_2017-10-29_23_11_15_146
>>
>> I'm not sure what's causing it to complain there. But, I'm on a plane
>> right now (which is why this is a top post, sorry) so I can't really dig
>> much more than that. I'll try to take a deeper look at things later when
>> I'm on solid ground. (hopefully someone will beat me to it by then though)
>>
>> I don't think that l3-agent trace is it, as the failure is coming from
>> the API.  It's actually a trace that's happening due to the async nature
>> of how the agent runs arping, fix is
>> https://review.openstack.org/#/c/507914/ b

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Tobias Urdin]

00 ERROR nova.api.openstack.compute.floating_ips 
return self.session.request(url, method, **kwargs)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 101, in 
inner
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
return wrapped(*args, **kwargs)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 703, in 
request
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
resp = send(**kwargs)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips   
File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 768, in 
_send_request
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
raise exceptions.ConnectTimeout(msg)
2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
ConnectTimeout: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out

Nova compute API properly running with wsgi under apache2? And nova metadata 
API running under the nova-api process?
Yea, must be otherwise they would fail to bind.

Best regards
Tobias

On 11/14/2017 09:28 AM, Tobias Urdin wrote:
Trying to trace this, tempest calls the POST /servers//action API 
endpoint for the nova compute api.

https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/floating_ips_client.py#L82

Nova then takes the requests and tries to do this floating ip association using 
the neutron server api.

http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/nova/nova-api.txt.gz

2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips 
[req-7f810cc7-a498-4bf4-b27e-8fc80d652785 42526a28b1a14c629b83908b2d75c647 
2493426e6a3c4253a60c0b7eb35cfe19 - default default] Unable to associate 
floating IP 172.24.5.17 to fixed IP 10.100.0.8 for instance 
d265626a-77c1-4d2f-8260-46abe548293e. Error: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out: ConnectTimeout: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out

Checking that timestamp in the neutron-server logs:
http://paste.openstack.org/show/626240/

We can see that during this timestamp right before at 23:12:30.377 and then 
after 23:12:35.611 everything seems to be doing fine.
So there is some connectivity issues to the neutron API from where the Nova API 
is running causing a timeout.

Now some more questions would be:

* Why is the return code 400? Are we being fooled or is it actually a 
connection timeout.
* Is the Neutron API stuck causing the failed connection? All talk are done 
over loopback so chance of a problem there is very low.
* Any firewall catching this? Not likely since the agent processes requests 
right before and after.

I can't find anything interesting in the overall other system logs that could 
explain that.
Back to the logs!

Best regards
Tobias


On 11/14/2017 08:35 AM, Tobias Urdin wrote:

Hello,

Same here, I will continue looking at this aswell.

Would be great if we could get some input from a neutron dev with good insight 
into the project.


Can we backtrace the timed out message from where it's thrown/returned.

Error: Request to 
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c 
timed out', u'code': 400}

I'm interested why we would get 400 code back, the floating ip operations 
should be async right so this would be the response from the API layer with 
information from the database that should

return more information.


Best regards

On 11/14/2017 07:41 AM, Arnaud MORIN wrote:
Hey,
Do you know if the bug appears on a specific Ubuntu / openstack version?
As far as I remember it was not related to the puppet branch?  I mean the bug 
is existing on master but also on newton puppet branches, right?

We are using Ubuntu in my company so we would love to see that continue ;)
I'll try to take a look again.

Cheers.

Le 14 nov. 2017 00:00, "Mohammed Naser" 
<mna...@vexxhost.com<mailto:mna...@vexxhost.com>> a écrit :
Hi,

Hope that everyone had safe travels and enjoyed their time at Sydney
(and those who weren't there enjoyed a bit of quiet time!).  I'm just
sending this email if anyone had a chance to look more into this (or
perhaps we can get some help if there are any Canonical folks on the
list?)

I would be really sad if we had to drop Ubuntu/Debian support because
we cannot test it.  I think there are a lot of users out there using
it!  I'd be more than happy to provide any assistance/information in
troubleshooting this.

Thank you,
Mohammed

On Thu, Nov 2, 2017 at 1:10 PM, Mohammed Naser 
<mna...@vexxhost.com

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Tobias Urdin]

Yea, I've been scavenging the logs for any kind of indicator on what
might have gone wrong but I can't see anything
related to a deadlock even though I'm very certain that's the issue but
don't know what's causing it.

Perhaps we will need to manually recreate this issue and then
troubleshoot it manually.
The apache2 mod_wsgi config should be OK according to the docs [1].

Best regards

[1]
http://modwsgi.readthedocs.io/en/develop/configuration-directives/WSGIDaemonProcess.html

On 11/14/2017 11:12 AM, Jens Harbott wrote:
> 2017-11-14 8:24 GMT+00:00 Tobias Urdin <tobias.ur...@crystone.com>:
>> Trying to trace this, tempest calls the POST /servers//action
>> API endpoint for the nova compute api.
>>
>> https://github.com/openstack/tempest/blob/master/tempest/lib/services/compute/floating_ips_client.py#L82
>>
>> Nova then takes the requests and tries to do this floating ip association
>> using the neutron server api.
>>
>> http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/nova/nova-api.txt.gz
>>
>> 2017-10-29 23:12:35.521 17800 ERROR nova.api.openstack.compute.floating_ips
>> [req-7f810cc7-a498-4bf4-b27e-8fc80d652785 42526a28b1a14c629b83908b2d75c647
>> 2493426e6a3c4253a60c0b7eb35cfe19 - default default] Unable to associate
>> floating IP 172.24.5.17 to fixed IP 10.100.0.8 for instance
>> d265626a-77c1-4d2f-8260-46abe548293e. Error: Request to
>> https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c
>> timed out: ConnectTimeout: Request to
>> https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c
>> timed out
>>
>> Checking that timestamp in the neutron-server logs:
>> http://paste.openstack.org/show/626240/
>>
>> We can see that during this timestamp right before at 23:12:30.377 and then
>> after 23:12:35.611 everything seems to be doing fine.
>> So there is some connectivity issues to the neutron API from where the Nova
>> API is running causing a timeout.
>>
>> Now some more questions would be:
>>
>> * Why is the return code 400? Are we being fooled or is it actually a
>> connection timeout.
>> * Is the Neutron API stuck causing the failed connection? All talk are done
>> over loopback so chance of a problem there is very low.
>> * Any firewall catching this? Not likely since the agent processes requests
>> right before and after.
>>
>> I can't find anything interesting in the overall other system logs that
>> could explain that.
>> Back to the logs!
> I'm pretty certain that this is a deadlock between nova and neutron,
> though I cannot put my finger on the exact spot yet. But looking at
> the neutron log that you extracted you can see that neutron indeed
> tries to give a successful answer to the fip request just after nova
> has given up waiting for it (seems the timeout is 30s here):
>
> 2017-10-29 23:12:35.932 18958 INFO neutron.wsgi
> [req-e737b7dd-ed9c-46a7-911b-eb77efe11aa8
> 42526a28b1a14c629b83908b2d75c647 2493426e6a3c4253a60c0b7eb35cfe19 -
> default default] 127.0.0.1 "PUT
> /v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c HTTP/1.1"
> status: 200  len: 746 time: 30.4427412
>
> Also, looking at
> http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/apache_config/10-nova_api_wsgi.conf.txt.gz
> is seems that nova-api is started with two processes and one thread,
> not sure if that means two processes with one thread each or only one
> thread total, anyway nova-api might be getting stuck there.
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet][qa][ubuntu][neutron] Xenial Neutron Timeouts

[Tobias Urdin]

I've been staring at this for almost an hour now going through all the logs and 
I can't really pin a point from

where that error message is generated. Cannot find any references for the timed 
out message that the API returns or the unable to associate part.


What I'm currently staring at is why would the instance fixed ip 172.24.5.17 be 
references as a network:router_gateway port in the OVS agent logs.

2017-10-29 
23:19:27.591
 11856 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 
[req-7274c6f7-18ef-420d-ad5a-9d0fe4eb35c6 - - - - -] Port 
053a625c-4227-41fb-9a26-45eda7bd2055 updated. Details: {'profile': {}, 
'network_qos_policy_id': None, 'qos_policy_id': None, 'allowed_address_pairs': 
[], 'admin_state_up': True, 'network_id': 
'f9647756-41ad-4ec5-af49-daefe410815e', 'segmentation_id': None, 'fixed_ips': 
[{'subnet_id': 'a31c7115-1f3e-4220-8bdb-981b6df2e18c', 'ip_address': 
'172.24.5.17'}], 'device_owner': u'network:router_gateway', 'physical_network': 
u'external', 'mac_address': 'fa:16:3e:3b:ec:c3', 'device': 
u'053a625c-4227-41fb-9a26-45eda7bd2055', 'port_security_enabled': False, 
'port_id': '053a625c-4227-41fb-9a26-45eda7bd2055', 'network_type': u'flat', 
'security_groups': []}

Anybody else seen anything interesting?

On 10/30/2017 11:08 PM, Brian Haley wrote:

On 10/30/2017 05:46 PM, Matthew Treinish wrote:


 From a quick glance at the logs my guess is that the issue is related
to this stack trace in the l3 agent logs:

http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/neutron/neutron-l3-agent.txt.gz?level=TRACE#_2017-10-29_23_11_15_146

I'm not sure what's causing it to complain there. But, I'm on a plane
right now (which is why this is a top post, sorry) so I can't really dig
much more than that. I'll try to take a deeper look at things later when
I'm on solid ground. (hopefully someone will beat me to it by then though)



I don't think that l3-agent trace is it, as the failure is coming from
the API.  It's actually a trace that's happening due to the async nature
of how the agent runs arping, fix is
https://review.openstack.org/#/c/507914/ but it only removes the log noise.

http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/neutron/neutron-server.txt.gz
has some tracebacks that look config related, possible missing DB table?
  But I haven't looked very closely.

-Brian




On October 31, 2017 1:25:55 AM GMT+04:00, Mohammed Naser
 wrote:

Hi everyone,

I'm looking for some help regarding an issue that we're having with
the Puppet OpenStack modules, we've had very inconsistent failures in
the Xenial with the following error:

 
http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/
 
http://logs.openstack.org/47/514347/1/check/puppet-openstack-integration-4-scenario001-tempest-ubuntu-xenial/ed5a657/logs/testr_results.html.gz
 Details: {u'message': u'Unable to associate floating IP
172.24.5.17   to fixed IP10.100.0.8 
  for instance
d265626a-77c1-4d2f-8260-46abe548293e. Error: Request to
https://127.0.0.1:9696/v2.0/floatingips/2e3fa334-d6ac-443c-b5ba-eeb521d6324c
timed out', u'code': 400}

At this point, we're at a bit of a loss.  I've tried my best in order
to find the root cause however we have not been able to do this.  It
was persistent enough that we elected to go non-voting for our Xenial
gates, however, with no fix ahead of us, I feel like this is a waste
of resources and we need to either fix this or drop CI for Ubuntu.  We
don't deploy on Ubuntu and most of the developers working on the
project don't either at this point, so we need a bit of resources.

If you're a user of Puppet on Xenial, we need your help!  Without any
resources going to fix this, we'd unfortunately have to drop support
for Ubuntu because of the lack of resources to maintain it (or
assistance).  We (Puppet OpenStack team) would be more than happy to
work together to fix this so pop-in at #puppet-openstack or reply to
this email and let's get this issue fixed.

Thanks,
Mohammed



OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] OpenStack Queens for Ubuntu 18.04 LTS

[Tobias Urdin]

I got started on kuryr-libnetwork but never finished the init/systemd scripts 
but all dependencies in control file should be ok.

I uploaded it here: https://github.com/tobias-urdin/deb-kuryr-libnetwork (not a 
working package!)

After fixing kuryr-libnetwork one can get starting packaging Zun.


For Qinling you might want kuryr-libkubernetes as well, but I'm unsure.

Best regards

On 04/27/2018 05:56 PM, Corey Bryant wrote:


On Fri, Apr 27, 2018 at 11:23 AM, Tobias Urdin 
<tobias.ur...@crystone.com<mailto:tobias.ur...@crystone.com>> wrote:

Hello,

I was very interested in packaging Zun for Ubuntu however I did not have the 
time to properly get started.

I was able to package kuryr-lib, I've uploaded it here for now 
https://github.com/tobias-urdin/deb-kuryr-lib


Would love to see both Zun and Qinling in Ubuntu to get a good grip on the 
container world :)

Best regards


Awesome Tobias. I can take a closer look next week if you'd like.

Thanks,
Corey

On 04/27/2018 04:59 PM, Corey Bryant wrote:
On Fri, Apr 27, 2018 at 10:20 AM, Hongbin Lu 
<hongbin...@gmail.com<mailto:hongbin...@gmail.com>> wrote:
Corey,

Thanks for the information. Would you clarify what is "working packages from 
the community"?

Best regards,
Hongbin

Sorry I guess that comment is probably a bit vague.

The OpenStack packages are open source like many other projects. They're Apache 
2 licensed and we gladly accept contributions. :)

This is a good starting point for working with the Ubuntu OpenStack packages:
https://wiki.ubuntu.com/OpenStack/CorePackages

If you or someone else were to provide package sources for zun that DTRT to 
create binary packages, and if they can test them, then I'd be happy to 
review/sponsor the Ubuntu and cloud-archive uploads.

Thanks,
Corey


On Fri, Apr 27, 2018 at 9:30 AM, Corey Bryant 
<corey.bry...@canonical.com<mailto:corey.bry...@canonical.com>> wrote:


On Fri, Apr 27, 2018 at 9:03 AM, Hongbin Lu 
<hongbin...@gmail.com<mailto:hongbin...@gmail.com>> wrote:
Hi Corey,

What are the requirements to include OpenStack Zun into the Ubuntu packages? We 
have a comprehensive installation guide [1] that are using by a lot of users 
when they were installing Zun. However, the missing of Ubuntu packages is 
inconvenient for our users. What the Zun team can help for adding Zun to Ubuntu.

[1] https://docs.openstack.org/zun/latest/install/index.html

Best regards,
Hongbin

Hi Hongbin,

If we were to get working packages from the community and commitment to test, 
I'd be happy to sponsor uploads to Ubuntu and backport to the cloud achive.

Thanks,
Corey

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack] OpenStack Queens for Ubuntu 18.04 LTS

[Tobias Urdin]

Hello,

I was very interested in packaging Zun for Ubuntu however I did not have the 
time to properly get started.

I was able to package kuryr-lib, I've uploaded it here for now 
https://github.com/tobias-urdin/deb-kuryr-lib


Would love to see both Zun and Qinling in Ubuntu to get a good grip on the 
container world :)

Best regards

On 04/27/2018 04:59 PM, Corey Bryant wrote:
On Fri, Apr 27, 2018 at 10:20 AM, Hongbin Lu 
<hongbin...@gmail.com<mailto:hongbin...@gmail.com>> wrote:
Corey,

Thanks for the information. Would you clarify what is "working packages from 
the community"?

Best regards,
Hongbin

Sorry I guess that comment is probably a bit vague.

The OpenStack packages are open source like many other projects. They're Apache 
2 licensed and we gladly accept contributions. :)

This is a good starting point for working with the Ubuntu OpenStack packages:
https://wiki.ubuntu.com/OpenStack/CorePackages

If you or someone else were to provide package sources for zun that DTRT to 
create binary packages, and if they can test them, then I'd be happy to 
review/sponsor the Ubuntu and cloud-archive uploads.

Thanks,
Corey


On Fri, Apr 27, 2018 at 9:30 AM, Corey Bryant 
<corey.bry...@canonical.com<mailto:corey.bry...@canonical.com>> wrote:


On Fri, Apr 27, 2018 at 9:03 AM, Hongbin Lu 
<hongbin...@gmail.com<mailto:hongbin...@gmail.com>> wrote:
Hi Corey,

What are the requirements to include OpenStack Zun into the Ubuntu packages? We 
have a comprehensive installation guide [1] that are using by a lot of users 
when they were installing Zun. However, the missing of Ubuntu packages is 
inconvenient for our users. What the Zun team can help for adding Zun to Ubuntu.

[1] https://docs.openstack.org/zun/latest/install/index.html

Best regards,
Hongbin

Hi Hongbin,

If we were to get working packages from the community and commitment to test, 
I'd be happy to sponsor uploads to Ubuntu and backport to the cloud achive.

Thanks,
Corey

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] [magnum] Magnum tempest fails with 400 bad request

[Tobias Urdin]

Hello,

I was interested in getting Magnum working in gate by getting @dms patch
fixed and merged [1].

The installation goes fine on Ubuntu and CentOS however the tempest
testing for Magnum fails on CentOS (it not available in Ubuntu).


It seems to be related to authentication against keystone but I don't
understand why, please see logs [2] [3]


[1] https://review.openstack.org/#/c/367012/

[2]
http://logs.openstack.org/12/367012/28/check/puppet-openstack-integration-4-scenario003-tempest-centos-7/3f5252b/logs/magnum/magnum-api.txt.gz#_2018-05-16_15_10_36_010

[3]
http://logs.openstack.org/12/367012/28/check/puppet-openstack-integration-4-scenario003-tempest-centos-7/3f5252b/


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [cloudkitty] configuration, deployment or packaging issue?

[Tobias Urdin]

Hello CloudKitty team,


I'm having an issue with this review not going through and being stuck after 
staring at it for a while now [1].

Is there any configuration[2] issue that are causing the error[3]? Or is the 
package broken?


Thanks for helping out!

Best regards


[1] https://review.openstack.org/#/c/569641/

[2] 
http://logs.openstack.org/41/569641/1/check/puppet-openstack-beaker-centos-7/ee4742c/logs/etc/cloudkitty/

[3] 
http://logs.openstack.org/41/569641/1/check/puppet-openstack-beaker-centos-7/ee4742c/logs/cloudkitty/processor.txt.gz
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cloudkitty] configuration, deployment or packaging issue?

[Tobias Urdin]

Hello,

Thanks Alex, I should probably improve my search-fu.

Is that commit in the RPM packages then I assume, so we need to ship a
metrics.yaml (which is kind of opinionated unless CloudKitty supplies a
default one)  and set the fetcher in the config file.


Perhaps somebody can confirm the above.

Best regards


On 06/19/2018 12:35 AM, Alex Schultz wrote:
> On Mon, Jun 18, 2018 at 4:08 PM, Tobias Urdin  
> wrote:
>> Hello CloudKitty team,
>>
>>
>> I'm having an issue with this review not going through and being stuck after
>> staring at it for a while now [1].
>>
>> Is there any configuration[2] issue that are causing the error[3]? Or is the
>> package broken?
>>
> Likely due to https://review.openstack.org/#/c/538256/ which appears
> to change the metrics.yaml format. It doesn't look backwards
> compatible so the puppet module probably needs updating.
>
>> Thanks for helping out!
>>
>> Best regards
>>
>>
>> [1] https://review.openstack.org/#/c/569641/
>>
>> [2]
>> http://logs.openstack.org/41/569641/1/check/puppet-openstack-beaker-centos-7/ee4742c/logs/etc/cloudkitty/
>>
>> [3]
>> http://logs.openstack.org/41/569641/1/check/puppet-openstack-beaker-centos-7/ee4742c/logs/cloudkitty/processor.txt.gz
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ovs] [neutron] openvswitch flows firewall driver

[Tobias Urdin]

Hello everybody,
I'm cross-posting this with operators list.

The openvswitch flows-based stateful firewall driver which uses the
conntrack support in Linux kernel >= 4.3 (iirc) has been
marked as experimental for several releases now, is there any
information about flaws in this and why it should not be used in production?

It's still marked as experimental or missing documentation in the
networking guide [1].

And to operators; is anybody running the OVS stateful firewall in
production? (firewall_driver = openvswitch)

Appreciate any feedback :)
Best regards

[1] https://docs.openstack.org/neutron/queens/admin/config-ovsfwdriver.html

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [tripleo][puppet] Hello all, puppet modules

[Tobias Urdin]

We are using them for one of our deployments and are working on moving
our other one to use the same modules :)

Best regards


On 06/04/2018 11:06 PM, Arnaud Morin wrote:
> Hey,
>
> OVH is also using them as well as some custom ansible playbooks to manage the 
> deployment. But as for red had, the configuration part is handled by puppet.
> We are also doing some upstream contribution from time to time.
>
> For us, the puppet modules are very stable and works very fine.
>
> Cheers,
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet-openstack][announce][debian] puppet-openstack now has full Debian support

[Tobias Urdin]

Great work Thomas!

We also got a good overview on some challenges and issues regarding
moving to python3 support for the Puppet modules in the future.
But we can easily state that the main work will lay with making sure
there is python3 supported distro packages.

Which you with Debian python3 support has also paved the way with.
Looking forward to all python3 release in the future, all aboard the train!

Best regards
Tobias

On 06/20/2018 04:25 PM, Thomas Goirand wrote:
> Dear Stackers,
>
> I am glad/overjoyed/jazzed to announce the global availability of
> puppet-openstack for Debian. Indeed, a few minutes ago, the CI turned
> all green for Debian:
>
> https://review.openstack.org/#/c/576416
>
> (note: the red one for CentOS is to be ignored, it looks like
> non-deterministic error)
>
> This is after 3 months of hard work, and more than 50 patches, sometimes
> on upstream code base (for example in Cinder, Sahara, and Neutron),
> often because of Python 3 or uwsgi/mod_wsgi related problems. Some of
> these patches aren't merged yet upstream, but are included in the Debian
> packages already. Also note that Debian fully supports SSL and ipv6
> endpoints.
>
> I'd like here to publicly thanks all of the puppet-openstack core
> reviewers for their help and enthusiasm. A big thanks to mnaser,
> tobasco, EmilienM and mwhahaha. Guys, you've been really awesome and
> helpful with me. Also a big thanks to these upstream helping with fixing
> these bits as explained above, and especially annp for fixing the
> neutron-rpc-server related problems, with the patch also pending reviews
> at: https://review.openstack.org/#/c/555608/
>
> All of these puppet modules are available directly in Debian in a
> packaged form. To get them, simply do:
>
> apt-get install openstack-puppet-modules
>
> in Debian Sid, or using the Debian backports repository at:
>
> http://stretch-queens.infomaniak.ch/debian
>
> Still to fix, is neutron-fwaas, which seems to not like either Python 3
> or using neutron-api over uwsgi (I'm not sure which of these yet).
> Upstream neutron developers are currently investigating this. For this
> reason, neutron firewall extension is currently disabled for the
> l3-agent, but will be reactivated as soon as a proper fix is found.
>
> Also, Ceph in Debian is currently a way behind (so we have to use
> upstream Debian repository for Stretch), as it lacks a proper Python 3
> support, and still no Luminous release uploaded to Sid. I intend to
> attempt to fix this, to get a chance to get this in time for Buster.
>
> Cheers,
>
> Thomas Goirand (zigo)
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Ubuntu problems + Help needed

[Tobias Urdin]

Hello everyone and a happy new year!

I will follow this thread up with some information about the tempest failure 
that occurs on Ubuntu.
Saw it happen on my recheck tonight and took some time now to check it out 
properly.

* Here is the job: 
http://logs.openstack.org/37/529837/1/check/puppet-openstack-integration-4-scenario003-tempest-ubuntu-xenial/84b60a7/

* The following test is failing but only sometimes: 
tempest.api.compute.servers.test_create_server.ServersTestManualDisk
http://logs.openstack.org/37/529837/1/check/puppet-openstack-integration-4-scenario003-tempest-ubuntu-xenial/84b60a7/job-output.txt.gz#_2018-01-07_01_56_31_072370

* Checking the nova API log is fails the request against neutron server
http://logs.openstack.org/37/529837/1/check/puppet-openstack-integration-4-scenario003-tempest-ubuntu-xenial/84b60a7/logs/nova/nova-api.txt.gz#_2018-01-07_01_46_47_301

So this is the call that times out: 
https://github.com/openstack/nova/blob/3800cf6ae2a1370882f39e6880b7df4ec93f4b93/nova/api/openstack/compute/attach_interfaces.py#L61

The timeout occurs at 01:46:47 but the first try is done at 01:46:17, checking 
the log 
http://logs.openstack.org/37/529837/1/check/puppet-openstack-integration-4-scenario003-tempest-ubuntu-xenial/84b60a7/logs/neutron/neutron-server.txt.gz
 and searching for "GET 
/v2.0/ports?device_id=285061f8-2e8e-4163-9534-9b02900a8887"

You can see that neutron-server reports all request as 200 OK, so what I think 
is that neutron-server performs the request properly but for some reason 
nova-api does not get the reply and hence the timeout.

This is where I get stuck because since I can see all requests coming in there 
is no real way of seeing the replies.
At the same time you can see nova-api and neutron-server are continously 
handling requests so they are working but just that reply that neutron-server 
should send to nova-api does not occur.

Does anybody have any clue to why? Otherwise I guess the only way is to start 
running the tests on a local machine until I get that issue, which does not 
occur regularly.

Maybe loop in the neutron and/or Canonical OpenStack team on this one.

Best regards
Tobias


____
From: Tobias Urdin <tobias.ur...@crystone.com>
Sent: Friday, December 22, 2017 2:44 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [puppet] Ubuntu problems + Help needed

Follow up, have been testing some integration runs on a tmp machine.

Had to fix the following:
* Ceph repo key E84AC2C0460F3994 perhaps introduced in [0]
* Run glance-manage db_sync (have not seen in integration tests)
* Run neutron-db-manage upgrade heads (have not seen in integration tests)
* Disable l2gw because of
https://bugs.launchpad.net/ubuntu/+source/networking-l2gw/+bug/1739779
   proposed temp fix until resolved as [1]

[0] https://review.openstack.org/#/c/507925/
[1] https://review.openstack.org/#/c/529830/

Best regards

On 12/22/2017 10:44 AM, Tobias Urdin wrote:
> Ignore that, seems like it's the networking-l2gw package that fails[0]
> Seems like it hasn't been packaged for queens yet[1] or more it seems
> like a release has not been cut for queens for networking-l2gw[2]
>
> Should we try to disable l2gw like done in[3] recently for CentOS?
>
> [0]
> http://logs.openstack.org/57/529657/2/check/puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ce6f987/logs/neutron/neutron-server.txt.gz#_2017-12-21_23_10_05_564
> [1]
> http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/queens_versions.html
> [2] https://git.openstack.org/cgit/openstack/networking-l2gw/refs/
> [3] https://review.openstack.org/#/c/529711/
>
>
> On 12/22/2017 10:19 AM, Tobias Urdin wrote:
>> Follow up on Alex[1] point. The db sync upgrade for neutron fails here[0].
>>
>> [0] http://paste.openstack.org/show/629628/
>>
>> On 12/22/2017 04:57 AM, Alex Schultz wrote:
>>>> Just a note, the queens repo is not currently synced in the infra so
>>>> the queens repo patch is failing on Ubuntu jobs. I've proposed adding
>>>> queens to the infra configuration to resolve this:
>>>> https://review.openstack.org/529670
>>>>
>>> As a follow up, the mirrors have landed and two of the four scenarios
>>> now pass.  Scenario001 is failing on ceilometer-api which was removed
>>> so I have a patch[0] to remove it. Scenario004 is having issues with
>>> neutron and the db looks to be very unhappy[1].
>>>
>>> Thanks,
>>> -Alex
>>>
>>> [0] https://review.openstack.org/529787
>>> [1] 
>>> http://logs.openstack.org/57/529657/2/check/puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ce6f987/logs/neutron/neutron-server.txt.gz#_2017-12-21_22_58_37_338
>>>
>>&g

Re: [openstack-dev] Debian OpenStack packages switching to Py3 for Queens

[Tobias Urdin]

On 02/15/2018 08:52 PM, Matthew Thode wrote:

> On 18-02-15 13:38:45, Matthew Thode wrote:
>> On 18-02-15 09:31:19, Thomas Goirand wrote:
>>> Hi,
>>>
>>> Since I'm getting some pressure from other DDs to actively remove Py2
>>> support from my packages, I'm very much considering switching all of the
>>> Debian packages for Queens to using exclusively Py3. I would have like
>>> to read some opinions about this. Is it a good time for such move? I
>>> hope it is, because I'd like to maintain as few Python package with Py2
>>> support at the time of Debian Buster freeze.
>>>
>>> Also, doing Queens, I've noticed that os-xenapi is still full of py2
>>> only stuff in os_xenapi/dom0. Can we get those fixes? Here's my patch:
>>>
>>> https://review.openstack.org/544809
>>>
>> Gentoo has Openstack packaged for both python2.7 and python3.(5,6) for
>> pike.  Queens will be the same for us at least, but I haven't had
>> problems with at least the core services running them all through
>> python3.x.
>>
> Edit: Everything BUT swift...
>
Shimming in with more of an operators view; since a lot of services are
deployed with wsgi in a webserver, mainly a lot of API services, having
a coordinated approach of moving
all those services over to py3 at the same time is appreciated.
Otherwise a lot of the services cannot be co-located on the same machine
and will probably break other projects, for example the Puppet modules.

Same goes for testing all those services in a single node CI test.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Ubuntu problems + Help needed

[Tobias Urdin]

Follow up on Alex[1] point. The db sync upgrade for neutron fails here[0].

[0] http://paste.openstack.org/show/629628/

On 12/22/2017 04:57 AM, Alex Schultz wrote:
>> Just a note, the queens repo is not currently synced in the infra so
>> the queens repo patch is failing on Ubuntu jobs. I've proposed adding
>> queens to the infra configuration to resolve this:
>> https://review.openstack.org/529670
>>
> As a follow up, the mirrors have landed and two of the four scenarios
> now pass.  Scenario001 is failing on ceilometer-api which was removed
> so I have a patch[0] to remove it. Scenario004 is having issues with
> neutron and the db looks to be very unhappy[1].
>
> Thanks,
> -Alex
>
> [0] https://review.openstack.org/529787
> [1] 
> http://logs.openstack.org/57/529657/2/check/puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ce6f987/logs/neutron/neutron-server.txt.gz#_2017-12-21_22_58_37_338
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Ubuntu problems + Help needed

[Tobias Urdin]

Ignore that, seems like it's the networking-l2gw package that fails[0]
Seems like it hasn't been packaged for queens yet[1] or more it seems
like a release has not been cut for queens for networking-l2gw[2]

Should we try to disable l2gw like done in[3] recently for CentOS?

[0]
http://logs.openstack.org/57/529657/2/check/puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ce6f987/logs/neutron/neutron-server.txt.gz#_2017-12-21_23_10_05_564
[1]
http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/queens_versions.html
[2] https://git.openstack.org/cgit/openstack/networking-l2gw/refs/
[3] https://review.openstack.org/#/c/529711/


On 12/22/2017 10:19 AM, Tobias Urdin wrote:
> Follow up on Alex[1] point. The db sync upgrade for neutron fails here[0].
>
> [0] http://paste.openstack.org/show/629628/
>
> On 12/22/2017 04:57 AM, Alex Schultz wrote:
>>> Just a note, the queens repo is not currently synced in the infra so
>>> the queens repo patch is failing on Ubuntu jobs. I've proposed adding
>>> queens to the infra configuration to resolve this:
>>> https://review.openstack.org/529670
>>>
>> As a follow up, the mirrors have landed and two of the four scenarios
>> now pass.  Scenario001 is failing on ceilometer-api which was removed
>> so I have a patch[0] to remove it. Scenario004 is having issues with
>> neutron and the db looks to be very unhappy[1].
>>
>> Thanks,
>> -Alex
>>
>> [0] https://review.openstack.org/529787
>> [1] 
>> http://logs.openstack.org/57/529657/2/check/puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ce6f987/logs/neutron/neutron-server.txt.gz#_2017-12-21_22_58_37_338
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] Ubuntu problems + Help needed

[Tobias Urdin]

Follow up, have been testing some integration runs on a tmp machine.

Had to fix the following:
* Ceph repo key E84AC2C0460F3994 perhaps introduced in [0]
* Run glance-manage db_sync (have not seen in integration tests)
* Run neutron-db-manage upgrade heads (have not seen in integration tests)
* Disable l2gw because of
https://bugs.launchpad.net/ubuntu/+source/networking-l2gw/+bug/1739779
   proposed temp fix until resolved as [1]

[0] https://review.openstack.org/#/c/507925/
[1] https://review.openstack.org/#/c/529830/

Best regards

On 12/22/2017 10:44 AM, Tobias Urdin wrote:
> Ignore that, seems like it's the networking-l2gw package that fails[0]
> Seems like it hasn't been packaged for queens yet[1] or more it seems
> like a release has not been cut for queens for networking-l2gw[2]
>
> Should we try to disable l2gw like done in[3] recently for CentOS?
>
> [0]
> http://logs.openstack.org/57/529657/2/check/puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ce6f987/logs/neutron/neutron-server.txt.gz#_2017-12-21_23_10_05_564
> [1]
> http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/queens_versions.html
> [2] https://git.openstack.org/cgit/openstack/networking-l2gw/refs/
> [3] https://review.openstack.org/#/c/529711/
>
>
> On 12/22/2017 10:19 AM, Tobias Urdin wrote:
>> Follow up on Alex[1] point. The db sync upgrade for neutron fails here[0].
>>
>> [0] http://paste.openstack.org/show/629628/
>>
>> On 12/22/2017 04:57 AM, Alex Schultz wrote:
>>>> Just a note, the queens repo is not currently synced in the infra so
>>>> the queens repo patch is failing on Ubuntu jobs. I've proposed adding
>>>> queens to the infra configuration to resolve this:
>>>> https://review.openstack.org/529670
>>>>
>>> As a follow up, the mirrors have landed and two of the four scenarios
>>> now pass.  Scenario001 is failing on ceilometer-api which was removed
>>> so I have a patch[0] to remove it. Scenario004 is having issues with
>>> neutron and the db looks to be very unhappy[1].
>>>
>>> Thanks,
>>> -Alex
>>>
>>> [0] https://review.openstack.org/529787
>>> [1] 
>>> http://logs.openstack.org/57/529657/2/check/puppet-openstack-integration-4-scenario004-tempest-ubuntu-xenial/ce6f987/logs/neutron/neutron-server.txt.gz#_2017-12-21_22_58_37_338
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [magnum] supported OS images and magnum spawn failures for Swarm and Kubernetes

[Tobias Urdin]

Hello,

I'm testing around with Magnum and have so far only had issues.
I've tried deploying Docker Swarm (on Fedora Atomic 27, Fedora Atomic 
28) and Kubernetes (on Fedora Atomic 27) and haven't been able to get it 
working.


Running Queens, is there any information about supported images? Is 
Magnum maintained to support Fedora Atomic still?
What is in charge of population the certificates inside the instances, 
because this seems to be the root of all issues, I'm not using Barbican 
but the x509keypair driver

is that the reason?

Perhaps I missed some documentation that x509keypair does not support 
what I'm trying to do?


I've seen the following issues:

Docker:
* Master does not start and listen on TCP because of certificate issues
dockerd-current[1909]: Could not load X509 key pair (cert: 
"/etc/docker/server.crt", key: "/etc/docker/server.key")


* Node does not start with:
Dependency failed for Docker Application Container Engine.
docker.service: Job docker.service/start failed with result 'dependency'.

Kubernetes:
* Master etcd does not start because /run/etcd does not exist
** When that is created it fails to start because of certificate
2018-08-03 12:41:16.554257 C | etcdmain: open 
/etc/etcd/certs/server.crt: no such file or directory


* Master kube-apiserver does not start because of certificate
unable to load server certificate: open 
/etc/kubernetes/certs/server.crt: no such file or directory


* Master heat script just sleeps forever waiting for port 8080 to become 
available (kube-apiserver) so it can never kubectl apply the final steps.


* Node does not even start and times out when Heat deploys it, probably 
because master never finishes


Any help is appreciated perhaps I've missed something crucial, I've not 
tested Kubernetes on CoreOS yet.


Best regards
Tobias

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] [PTL] [Election] PTL candidacy for the Stein cycle

[Tobias Urdin]

Hello Stackers,

I'm submitting myself as PTL candidate for the Puppet OpenStack project. [0]

I've been active in the OpenStack community since late 2014 early 2015
and have had a lot of focus on the
Puppet OpenStack project since about 2016. I've been a core reviewer for
about five months now and it's
been really cool to be able to give something back to the community.

We have had a lot of progress this cycle.

* Remove a lot of deprecate parameters
* Improved testing of Puppet 5
* Added Debian 9 support (Python 3 only)
* Added Ubuntu 18.04 Bionic support
* Fixed some bugs
* Moved to more usage of the shared openstacklib resources
* Added neutron-dynamic-routing support
* Added horizon dashboard installation support
* Changed keystone to use port 5000 and deprecated usage of port 35357
(still deploys both)

I could ramble up a lot more in that list but I really think we've done
a good job but we still have some major things
moving forward that we'll have to work on. Here is some major things I
think we'll need to work on or discuss.

* Python 3 will be a big one, I know people are working on Fedora for
testing here, but we also have Debian9 here
   which is python3-only so thanks to Thomas (zigo) we have somebody
that has paved the way here.

* Puppet 5 data types for parameters and removing validate_* functions
is a big one which we also have an open blueprint
   and PoC for but will require a lot of interaction with the TripleO
team. [1] [2]

* CI stability and maintenance will be a reoccurring thing we'll need to
focus on.

* Puppet providers are usually slow due to CLI utilies, we need to work
together to improve the performance of
   the CLI tooling or consider the move to API calls, this has been up
before but there hasn't been anybody there
   that has sponsored such work.

I want to really thank all of you for your huge amounts of work, all
across the OpenStack board and the Puppet OpenStack team.
Thank you for considering me.

Best regards
Tobias (tobasco @ IRC)

[0] https://review.openstack.org/#/c/587372/
[1] https://review.openstack.org/#/c/568929/
[2] https://review.openstack.org/#/c/569566/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] [ptg] Planning for PTG

[Tobias Urdin]

Hello all Puppeters,

We are approaching the PTG in Denver and want to remind everybody to make sure 
you book your accommodations because they will run out.
I've created an etherpad here [1] where you can add any topic you want to 
discuss.

I will not be able to attend the PTG, if we get any topics that needs 
discussion we need somebody that could moderate the discussions.
Please fill in the etherpad if you will be attending, it would be great if 
anybody that attended could take the moderation duty if we have
topics and require a room booking.


There will also be team photos if people are interested more information will 
follow from the PTG organizers about that.


Best regards
Tobias

[1] https://etherpad.openstack.org/p/puppet-ptg-stein​

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] migrating to storyboard

[Tobias Urdin]

Hello all incredible Puppeters,

I've tested setting up an Storyboard instance and test migrated 
puppet-ceph and it went without any issues there using the documentation 
[1] [2]

with just one minor issue during the SB setup [3].

My goal is that we will be able to swap to Storyboard during the Stein 
cycle but considering that we have a low activity on
bugs my opinion is that we could do this swap very easily anything soon 
as long as everybody is in favor of it.


Please let me know what you think about moving to Storyboard?
If everybody is in favor of it we can request a migration to infra 
according to documentation [2].


I will continue to test the import of all our project while people are 
collecting their thoughts and feedback :)


Best regards
Tobias

[1] https://docs.openstack.org/infra/storyboard/install/development.html
[2] https://docs.openstack.org/infra/storyboard/migration.html
[3] It failed with an error about launchpadlib not being installed, 
solved with `tox -e venv pip install launchpadlib`


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] migrating to storyboard

[Tobias Urdin]

Hello Kendall,

Thanks for your reply, that sounds awesome!
We can then dig around and see how everything looks when all project 
bugs are imported to stories.


I see no issues with being able to move to Storyboard anytime soon if 
the feedback for

moving is positive.

Best regards
Tobias

On 08/14/2018 09:06 PM, Kendall Nelson wrote:

Hello!

The error you hit can be resolved by adding launchpadlib to your 
tox.ini if I recall correctly..


also, if you'd like, I can run a test migration of puppet's launchpad 
projects into our storyboard-dev db (where I've done a ton of other 
test migrations) if you want to see how it looks/works with a larger 
db. Just let me know and I can kick it off.


As for a time to migrate, if you all are good with it, we usually 
schedule for Friday's so there is even less activity. Its a small 
project config change and then we just need an infra core to kick off 
the script once the change merges.


-Kendall (diablo_rojo)

On Tue, Aug 14, 2018 at 9:33 AM Tobias Urdin <mailto:tobias.ur...@binero.se>> wrote:


Hello all incredible Puppeters,

I've tested setting up an Storyboard instance and test migrated
puppet-ceph and it went without any issues there using the
documentation
[1] [2]
with just one minor issue during the SB setup [3].

My goal is that we will be able to swap to Storyboard during the
Stein
cycle but considering that we have a low activity on
bugs my opinion is that we could do this swap very easily anything
soon
as long as everybody is in favor of it.

Please let me know what you think about moving to Storyboard?
If everybody is in favor of it we can request a migration to infra
according to documentation [2].

I will continue to test the import of all our project while people
are
collecting their thoughts and feedback :)

Best regards
Tobias

[1]
https://docs.openstack.org/infra/storyboard/install/development.html
[2] https://docs.openstack.org/infra/storyboard/migration.html
<https://docs.openstack.org/infra/storyboard/migration.html>
[3] It failed with an error about launchpadlib not being installed,
solved with `tox -e venv pip install launchpadlib`

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [puppet] migrating to storyboard

[Tobias Urdin]

Hello Kendall,

I went through the list of projects [1] and could only really see two 
things.


1) puppet-rally and puppet-openstack-guide is missing

2) We have some support projects which doesn't really need bug tracking, 
where some others do.
    You can remove puppet-openstack-specs and 
puppet-openstack-cookiecutter all others would be

    nice to still have left so we can track bugs. [2]

Best regards
Tobias

[1] https://storyboard-dev.openstack.org/#!/project_group/60 
<https://storyboard-dev.openstack.org/#%21/project_group/60>
[2] Keeping puppet-openstack-integration (integration testing) and 
puppet-openstack_spec_helper (helper for testing).
  These two usually has a lot of changes so would be good to be 
able to track them.


On 08/16/2018 09:40 PM, Kendall Nelson wrote:

Hey :)

I created all the puppet openstack repos in the storyboard-dev 
envrionment and made a project group[1]. I am struggling a bit with 
finding all of your launchpad projects to perform the migrations 
through, can you share a list of all of them?


-Kendall (diablo_rojo)

[1] https://storyboard-dev.openstack.org/#!/project_group/60 
<https://storyboard-dev.openstack.org/#%21/project_group/60>


On Wed, Aug 15, 2018 at 12:08 AM Tobias Urdin <mailto:tobias.ur...@binero.se>> wrote:


Hello Kendall,

Thanks for your reply, that sounds awesome!
We can then dig around and see how everything looks when all
project bugs are imported to stories.

I see no issues with being able to move to Storyboard anytime soon
if the feedback for
moving is positive.

Best regards

Tobias


On 08/14/2018 09:06 PM, Kendall Nelson wrote:

Hello!

The error you hit can be resolved by adding launchpadlib to your
tox.ini if I recall correctly..

also, if you'd like, I can run a test migration of puppet's
launchpad projects into our storyboard-dev db (where I've done a
ton of other test migrations) if you want to see how it
looks/works with a larger db. Just let me know and I can kick it
off.

As for a time to migrate, if you all are good with it, we usually
schedule for Friday's so there is even less activity. Its a small
project config change and then we just need an infra core to kick
off the script once the change merges.

-Kendall (diablo_rojo)

On Tue, Aug 14, 2018 at 9:33 AM Tobias Urdin
mailto:tobias.ur...@binero.se>> wrote:

Hello all incredible Puppeters,

I've tested setting up an Storyboard instance and test migrated
puppet-ceph and it went without any issues there using the
documentation
[1] [2]
with just one minor issue during the SB setup [3].

My goal is that we will be able to swap to Storyboard during
the Stein
cycle but considering that we have a low activity on
bugs my opinion is that we could do this swap very easily
anything soon
as long as everybody is in favor of it.

Please let me know what you think about moving to Storyboard?
If everybody is in favor of it we can request a migration to
infra
according to documentation [2].

I will continue to test the import of all our project while
people are
collecting their thoughts and feedback :)

Best regards
Tobias

[1]
https://docs.openstack.org/infra/storyboard/install/development.html
[2] https://docs.openstack.org/infra/storyboard/migration.html
[3] It failed with an error about launchpadlib not being
installed,
solved with `tox -e venv pip install launchpadlib`


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] Puppet weekly recap - week 33

[Tobias Urdin]

Hello all Puppeteers!

Welcome to the weekly Puppet recap for week 33.
This is a weekly overview of what has changed in the Puppet OpenStack 
project the past week.



CHANGES


Changes in all modules
---
* Removed PE requirement from metadata.json
* Prepared Rocky RC1

Aodh
---
* Improved restarting Apache
   Changed so that Apache more accurately restarts services on 
configuration changes.


Glance
-
* Configure access_key and secret_key as secrets
* Fixed glance_image provider
   Stopped working after os_algo, os_hash_value and os_hidden was 
introduced.


Heat
--
* Improved restarting Apache
   Changed so that Apache more accurately restarts services on 
configuration changes.


Horizon
--
* Add wsgi_processes and wsgi_threads to horizon init
* apache wsgi: Exchange defaults for workers and threads
   Default values for Apache WSGI workers and threads changed.

Manila
-
* Support manila-api deployment with Apache WSGI
  You can now deploy manila-api under Apache WSGI

Murano
--
* Deprecated auth_uri option

Nova
---
* Configure access_key and secret_key as secrets

Puppet-OpenStack-Integration
-
* Test bgp-dragent in scenario004
   Now has full testing for the BGP agent provided by 
neutron-dynamic-routing

* Fix configure_facts.sh for RDO mirrors
* Run metadata-json-lint test in lint job
   Now runs metadata-json-lint in puppet-lint jobs if a metadata.json 
file exists.

* Test Sahara API with WSGI
   Sahara is now tested with API running under Apache WSGI

Puppet-OpenStack-Guide
--
* Updated latest RC1 version on release page

Panko

* Restart API also when run with Apache
   Correctly restart Apache on configuration changes

Sahara
--
* Add Sahara API WSGI support
  The Sahara API can now be deployed with Apache WSGI


SPECS

None.


OTHER

* We have submitted a review for releasing RC1 of all modules 
https://review.openstack.org/#/c/592584/

* We have started to take a look at migrating to Storyboard
   I have posted an email on the mailing list, please leave your 
feedback if you have any.


Interested in knowing what's up? Want to help or get help? See our 
etherpad https://etherpad.openstack.org/p/puppet-openstack-rocky
Or maybe you have some awesome ideas for next release? Let us know 
https://etherpad.openstack.org/p/puppet-openstack-stein




Wishing you all a great weekend!

Best regards
Tobias

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum] supported OS images and magnum spawn failures for Swarm and Kubernetes

[Tobias Urdin]

Now with Fedora 26 I have etcd available but etcd fails.

[root@swarm-u2rnie4d4ik6-master-0 ~]# /usr/bin/etcd 
--name="${ETCD_NAME}" --data-dir="${ETCD_DATA_DIR}" 
--listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}" --debug
2018-08-23 14:34:15.596516 E | etcdmain: error verifying flags, 
--advertise-client-urls is required when --listen-client-urls is set 
explicitly. See 'etcd --help'.
2018-08-23 14:34:15.596611 E | etcdmain: When listening on specific 
address(es), this etcd process must advertise accessible url(s) to each 
connected client.


There is a issue where the --advertise-client-urls and TLS --cert-file 
and --key-file is not passed in the systemd file, changing this to:
/usr/bin/etcd --name="${ETCD_NAME}" --data-dir="${ETCD_DATA_DIR}" 
--listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}" 
--advertise-client-urls="${ETCD_ADVERTISE_CLIENT_URLS}" 
--cert-file="${ETCD_PEER_CERT_FILE}" --key-file="${ETCD_PEER_KEY_FILE}"


Makes it work, any thoughts?

Best regards
Tobias

On 08/23/2018 03:54 PM, Tobias Urdin wrote:
Found the issue, I assume I have to use Fedora Atomic 26 until Rocky 
where I can start using Fedora Atomic 27.

Will Fedora Atomia 28 be supported for Rocky?

https://bugs.launchpad.net/magnum/+bug/1735381 (Run etcd and flanneld 
in system containers, In Fedora Atomic 27 etcd and flanneld are 
removed from the base image.)
https://review.openstack.org/#/c/524116/ (Run etcd and flanneld in a 
system container)


Still wondering about the "The Parameter (nodes_affinity_policy) was 
not provided" when using Mesos + Ubuntu?


Best regards
Tobias

On 08/23/2018 02:56 PM, Tobias Urdin wrote:

Thanks for all of your help everyone,

I've been busy with other thing but was able to pick up where I left 
regarding Magnum.
After fixing some issues I have been able to provision a working 
Kubernetes cluster.


I'm still having issues with getting Docker Swarm working, I've tried 
with both Docker and flannel as the networking layer but
none of these works. After investigating the issue seems to be that 
etcd.service is not installed (unit file doesn't exist) so the master
doesn't work, the minion swarm node is provisioned but cannot join 
the cluster because there is no etcd.


Anybody seen this issue before? I've been digging through all 
cloud-init logs and cannot see anything that would cause this.


I also have another separate issue, when provisioning using the 
magnum-ui in Horizon and selecting ubuntu with Mesos I get the error
"The Parameter (nodes_affinity_policy) was not provided". The 
nodes_affinity_policy do have a default value in magnum.conf so I'm 
starting

to think this might be an issue with the magnum-ui dashboard?

Best regards
Tobias

On 08/04/2018 06:24 PM, Joe Topjian wrote:
We recently deployed Magnum and I've been making my way through 
getting both Swarm and Kubernetes running. I also ran into some 
initial issues. These notes may or may not help, but thought I'd 
share them in case:


* We're using Barbican for SSL. I have not tried with the internal 
x509keypair.


* I was only able to get things running with Fedora Atomic 27, 
specifically the version used in the Magnum docs: 
https://docs.openstack.org/magnum/latest/install/launch-instance.html


Anything beyond that wouldn't even boot in my cloud. I haven't dug 
into this.


* Kubernetes requires a Cluster Template to have a label of 
cert_manager_api=true set in order for the cluster to fully come up 
(at least, it didn't work for me until I set this).


As far as troubleshooting methods go, check the cloud-init logs on 
the individual instances to see if any of the "parts" have failed to 
run. Manually re-run the parts on the command-line to get a better 
idea of why they failed. Review the actual script, figure out the 
variable interpolation and how it relates to the Cluster Template 
being used.


Eventually I was able to get clusters running with the stock 
driver/templates, but wanted to tune them in order to better fit in 
our cloud, so I've "forked" them. This is in no way a slight against 
the existing drivers/templates nor do I recommend doing this until 
you reach a point where the stock drivers won't meet your needs. But 
I mention it because it's possible to do and it's not terribly hard. 
This is still a work-in-progress and a bit hacky:


https://github.com/cybera/magnum-templates

Hope that helps,
Joe

On Fri, Aug 3, 2018 at 6:46 AM, Tobias Urdin <mailto:tobias.ur...@binero.se>> wrote:


Hello,

I'm testing around with Magnum and have so far only had issues.
I've tried deploying Docker Swarm (on Fedora Atomic 27, Fedora
Atomic 28) and Kubernetes (on Fedora Atomic 27) and haven't been
able to get it working.

Running Queens, is there any information about supported images?
Is Magnum maintained to support Fedora Atomic still?
What is in 

Re: [openstack-dev] [magnum] supported OS images and magnum spawn failures for Swarm and Kubernetes

[Tobias Urdin]

Found the issue, I assume I have to use Fedora Atomic 26 until Rocky 
where I can start using Fedora Atomic 27.

Will Fedora Atomia 28 be supported for Rocky?

https://bugs.launchpad.net/magnum/+bug/1735381 (Run etcd and flanneld in 
system containers, In Fedora Atomic 27 etcd and flanneld are removed 
from the base image.)
https://review.openstack.org/#/c/524116/ (Run etcd and flanneld in a 
system container)


Still wondering about the "The Parameter (nodes_affinity_policy) was not 
provided" when using Mesos + Ubuntu?


Best regards
Tobias

On 08/23/2018 02:56 PM, Tobias Urdin wrote:

Thanks for all of your help everyone,

I've been busy with other thing but was able to pick up where I left 
regarding Magnum.
After fixing some issues I have been able to provision a working 
Kubernetes cluster.


I'm still having issues with getting Docker Swarm working, I've tried 
with both Docker and flannel as the networking layer but
none of these works. After investigating the issue seems to be that 
etcd.service is not installed (unit file doesn't exist) so the master
doesn't work, the minion swarm node is provisioned but cannot join the 
cluster because there is no etcd.


Anybody seen this issue before? I've been digging through all 
cloud-init logs and cannot see anything that would cause this.


I also have another separate issue, when provisioning using the 
magnum-ui in Horizon and selecting ubuntu with Mesos I get the error
"The Parameter (nodes_affinity_policy) was not provided". The 
nodes_affinity_policy do have a default value in magnum.conf so I'm 
starting

to think this might be an issue with the magnum-ui dashboard?

Best regards
Tobias

On 08/04/2018 06:24 PM, Joe Topjian wrote:
We recently deployed Magnum and I've been making my way through 
getting both Swarm and Kubernetes running. I also ran into some 
initial issues. These notes may or may not help, but thought I'd 
share them in case:


* We're using Barbican for SSL. I have not tried with the internal 
x509keypair.


* I was only able to get things running with Fedora Atomic 27, 
specifically the version used in the Magnum docs: 
https://docs.openstack.org/magnum/latest/install/launch-instance.html


Anything beyond that wouldn't even boot in my cloud. I haven't dug 
into this.


* Kubernetes requires a Cluster Template to have a label of 
cert_manager_api=true set in order for the cluster to fully come up 
(at least, it didn't work for me until I set this).


As far as troubleshooting methods go, check the cloud-init logs on 
the individual instances to see if any of the "parts" have failed to 
run. Manually re-run the parts on the command-line to get a better 
idea of why they failed. Review the actual script, figure out the 
variable interpolation and how it relates to the Cluster Template 
being used.


Eventually I was able to get clusters running with the stock 
driver/templates, but wanted to tune them in order to better fit in 
our cloud, so I've "forked" them. This is in no way a slight against 
the existing drivers/templates nor do I recommend doing this until 
you reach a point where the stock drivers won't meet your needs. But 
I mention it because it's possible to do and it's not terribly hard. 
This is still a work-in-progress and a bit hacky:


https://github.com/cybera/magnum-templates

Hope that helps,
Joe

On Fri, Aug 3, 2018 at 6:46 AM, Tobias Urdin <mailto:tobias.ur...@binero.se>> wrote:


Hello,

I'm testing around with Magnum and have so far only had issues.
I've tried deploying Docker Swarm (on Fedora Atomic 27, Fedora
Atomic 28) and Kubernetes (on Fedora Atomic 27) and haven't been
able to get it working.

Running Queens, is there any information about supported images?
Is Magnum maintained to support Fedora Atomic still?
What is in charge of population the certificates inside the
instances, because this seems to be the root of all issues, I'm
not using Barbican but the x509keypair driver
is that the reason?

Perhaps I missed some documentation that x509keypair does not
support what I'm trying to do?

I've seen the following issues:

Docker:
* Master does not start and listen on TCP because of certificate
issues
dockerd-current[1909]: Could not load X509 key pair (cert:
"/etc/docker/server.crt", key: "/etc/docker/server.key")

* Node does not start with:
Dependency failed for Docker Application Container Engine.
docker.service: Job docker.service/start failed with result
'dependency'.

Kubernetes:
* Master etcd does not start because /run/etcd does not exist
** When that is created it fails to start because of certificate
2018-08-03 12:41:16.554257 C | etcdmain: open
/etc/etcd/certs/server.crt: no such file or directory

* Master kube-apiserver does not start because of certificate
unable to load server certificate: open
/etc

[openstack-dev] [puppet] Puppet weekly recap - week 34

[Tobias Urdin]

Hello all Puppeteers!

Welcome to the weekly Puppet recap for week 34.
This is a weekly overview of what has changed in the Puppet OpenStack 
project the past week.


CHANGES
===

We haven't had much changes this week, mostly CI fixes due to changes in 
packaging.


* We've merged all stable/rocky related changes except for Keystone [1] [2]
** This is blocked by packaging issue [3] because we dont update 
packages before runs in the beaker tests.

** Please review [4] and let us know what you think.
** This is also blocking this [5]
* Fixed puppet-ovn to make sure OVS bridge is created before setting 
mac-table-size [6]


[1] https://review.openstack.org/#/c/593787/
[2] https://review.openstack.org/#/c/593786/
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1620221
[4] https://review.openstack.org/#/c/595370/
[5] https://review.openstack.org/#/c/589877/
[6] https://review.openstack.org/#/c/594128/

REVIEWS
==

We have some open changes that needs reviews.

* Update packages after adding repos 
https://review.openstack.org/#/c/595370/
* Make vlan_transparent in neutron.conf configurable 
https://review.openstack.org/#/c/591899/
* neutron-dynamic-routing wrong package for Debian 
https://review.openstack.org/#/c/594058/ (and backports)
* Add workers to magnum api and conductor 
https://review.openstack.org/#/c/595228/

* Correct default number of threads https://review.openstack.org/#/c/591493/
* Deprecate unused notify_on_api_faults parameter 
https://review.openstack.org/#/c/593034/
* Resolve duplicate declaration with split of api / metadata wsgi 
https://review.openstack.org/#/c/595523/


SPECS
=
No new specs, only one open spec for review.

* Add parameter data types spec https://review.openstack.org/#/c/568929/

OTHER
=

* No new progress on the Storyboard migration, we will continue letting 
you know once we have more details about dates.
* Going to the PTG? We have some cores that will be there, make sure you 
say hi! [7]
** We dont have any planned talks or discussions and therefore dont need 
any session or a moderator, but we are always available if you need us 
on IRC at #puppet-openstack


* Interested in the current status for Rocky? See [8] or maybe you want 
to plan some awesome new cool thing then...
** Start planning Stein now [9] and let us know! We would love any new 
contributors with new cool ideas!


* We should have a walk through with abandoning old open changes, if 
anybody is interested in helping with such an effort, please let me know.


[7] https://etherpad.openstack.org/p/puppet-ptg-stein
[8] https://etherpad.openstack.org/p/puppet-openstack-rocky
[9] https://etherpad.openstack.org/p/puppet-openstack-stein

Wishing you all a great weekend!

Best regards
Tobias (tobias-urdin @ IRC)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Openstack-operators] [puppet] migrating to storyboard

[Tobias Urdin]

Hello Kendall,

I think you can just leave them in the group then, at your convenience.
If they are there we can start using them if so.

Best regards
Tobias

On 08/17/2018 11:08 PM, Kendall Nelson wrote:



On Fri, Aug 17, 2018 at 12:15 AM Tobias Urdin <mailto:tobias.ur...@binero.se>> wrote:


Hello Kendall,

I went through the list of projects [1] and could only really see
two things.

1) puppet-rally and puppet-openstack-guide is missing

I had created the projects, but missed adding them to the group. They 
should be there now :)


2) We have some support projects which doesn't really need bug
tracking, where some others do.
    You can remove puppet-openstack-specs and
puppet-openstack-cookiecutter all others would be
    nice to still have left so we can track bugs. [2]

i can remove them from the group if you want, but I don't think I can 
delete the projects entirely.


Best regards
Tobias

[1] https://storyboard-dev.openstack.org/#!/project_group/60
<https://storyboard-dev.openstack.org/#%21/project_group/60>
[2] Keeping puppet-openstack-integration (integration testing) and
puppet-openstack_spec_helper (helper for testing).
  These two usually has a lot of changes so would be good to
be able to track them.


On 08/16/2018 09:40 PM, Kendall Nelson wrote:

Hey :)

I created all the puppet openstack repos in the storyboard-dev
envrionment and made a project group[1]. I am struggling a bit
with finding all of your launchpad projects to perform the
migrations through, can you share a list of all of them?

-Kendall (diablo_rojo)

[1] https://storyboard-dev.openstack.org/#!/project_group/60
<https://storyboard-dev.openstack.org/#%21/project_group/60>

On Wed, Aug 15, 2018 at 12:08 AM Tobias Urdin
mailto:tobias.ur...@binero.se>> wrote:

Hello Kendall,

Thanks for your reply, that sounds awesome!
We can then dig around and see how everything looks when all
project bugs are imported to stories.

I see no issues with being able to move to Storyboard anytime
soon if the feedback for
moving is positive.

Best regards

Tobias


On 08/14/2018 09:06 PM, Kendall Nelson wrote:

Hello!

The error you hit can be resolved by adding launchpadlib to
your tox.ini if I recall correctly..

also, if you'd like, I can run a test migration of puppet's
launchpad projects into our storyboard-dev db (where I've
done a ton of other test migrations) if you want to see how
it looks/works with a larger db. Just let me know and I can
kick it off.

As for a time to migrate, if you all are good with it, we
usually schedule for Friday's so there is even less
activity. Its a small project config change and then we just
need an infra core to kick off the script once the change
merges.

-Kendall (diablo_rojo)

On Tue, Aug 14, 2018 at 9:33 AM Tobias Urdin
mailto:tobias.ur...@binero.se>> wrote:

Hello all incredible Puppeters,

I've tested setting up an Storyboard instance and test
migrated
puppet-ceph and it went without any issues there using
the documentation
[1] [2]
with just one minor issue during the SB setup [3].

My goal is that we will be able to swap to Storyboard
during the Stein
cycle but considering that we have a low activity on
bugs my opinion is that we could do this swap very
easily anything soon
as long as everybody is in favor of it.

Please let me know what you think about moving to
Storyboard?
If everybody is in favor of it we can request a
migration to infra
according to documentation [2].

I will continue to test the import of all our project
while people are
collecting their thoughts and feedback :)

Best regards
Tobias

[1]
https://docs.openstack.org/infra/storyboard/install/development.html
[2]
https://docs.openstack.org/infra/storyboard/migration.html
[3] It failed with an error about launchpadlib not being
installed,
solved with `tox -e venv pip install launchpadlib`


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
http://lists.openstack.org

Re: [openstack-dev] [neutron] neutron ipv6 radvd sends out link-local or nothing as def gw (L3 HA issue?)

[Tobias Urdin]

Ok, so the issue here seems to be that I have a L3 HA router with SLAAC, 
both the active and standby router will
configure the SLAAC obtained address causing a conflict since both side 
share the same MAC address.


Is there any workaround for this? Should SLAAC even be enabled for 
interfaces on the standby router?


Best regards
Tobias

On 08/20/2018 11:37 AM, Tobias Urdin wrote:

Forgot [neutron] tag.

On 08/20/2018 11:36 AM, Tobias Urdin wrote:

Hello,

Note: before reading, this router was a regular router but was then 
disable, changed ha=true so it's now a L3 HA router, then it was 
enabled again.

CC openstack-dev for help or feedback if it's a possible bug.

I've been testing around with IPv6 and overall the experience has 
been positive but I've met some weird issue that I cannot put my head 
around.
So this is a neutron L3 router with an outside interface with a ipv4 
and ipv6 from the provider network and one inside interface for ipv4 
and one inside interface for ipv6.


The instances for some reason get's there default gateway as the ipv6 
link-local (in fe80::/10) from the router with SLAAC and radvd.


(. is provider network, . is inside network, they are 
masked so don't pay attention to the number per se)


*interfaces inside router:*
15: ha-9bde1bb1-bd:  mtu 1450 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:05:80:32 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.7/18 brd 169.254.255.255 scope global ha-9bde1bb1-bd
   valid_lft forever preferred_lft forever
    inet 169.254.0.1/24 scope global ha-9bde1bb1-bd
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe05:8032/64 scope link
   valid_lft forever preferred_lft forever
19: qg-86e465f6-33:  mtu 1500 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:3b:8b:a5 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/22 scope global qg-86e465f6-33
   valid_lft forever preferred_lft forever
    inet6 :::f/64 scope global nodad
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe3b:8ba5/64 scope link nodad
   valid_lft forever preferred_lft forever
1168: qr-5be04815-68:  mtu 1450 
qdisc noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:c3:85:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.1/24 scope global qr-5be04815-68
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec3:85bd/64 scope link
   valid_lft forever preferred_lft forever
1169: qr-7fad6b1b-c9:  mtu 1450 
qdisc noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:66:de:a8 brd ff:ff:ff:ff:ff:ff
    inet6 ::0:1::1/64 scope global nodad
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe66:dea8/64 scope link
   valid_lft forever preferred_lft forever

I get this error messages in dmesg on the network node:
[581085.858869] IPv6: qr-5be04815-68: IPv6 duplicate address 
::0:1:f816:3eff:fec3:85bd detected!
[581085.997497] IPv6: qr-7fad6b1b-c9: IPv6 duplicate address 
::0:1:f816:3eff:fe66:dea8 detected!
[581142.869939] IPv6: qr-5be04815-68: IPv6 duplicate address 
::0:1:f816:3eff:fec3:85bd detected!
[581143.182371] IPv6: qr-7fad6b1b-c9: IPv6 duplicate address 
::0:1:f816:3eff:fe66:dea8 detected!


*radvd:*
interface qr-7fad6b1b-c9
{
   AdvSendAdvert on;
   MinRtrAdvInterval 30;
   MaxRtrAdvInterval 100;

   AdvLinkMTU 1450;

   RDNSS  2001:4860:4860::  {};

   prefix ::0:1::/64
   {
    AdvOnLink on;
    AdvAutonomous on;
   };
};

*inside instance:*
ipv4 = 192.168.199.7
ipv6 = ::0:1:f816:3eff:fe29:723d/64 (from radvd SLAAC)

I can ping ipv4 gateway 192.168.199.1 and internet over ipv4.
I can ping ipv6 gateway ::0:1::1 but I can't ping the internet

checking the ipv6 routing table on my instance I either get no 
default gateway at all or I get a default gateway to a fe80::/10 
link-local address.

IIRC this worked before I changed the router to a L3 HA router.

Appreciate any feedback!

Best regards
Tobias




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [neutron] neutron ipv6 radvd sends out link-local or nothing as def gw (L3 HA issue?)

[Tobias Urdin]

Continuing forward, these patches should've fixed that 
https://review.openstack.org/#/q/topic:bug/1667756+(status:open+OR+status:merged)

I'm on Queens.

The two inside interfaces on the backup router:
[root@controller2 ~]# ip netns exec 
qrouter-0775785e-a93a-4501-917b-be92ff03f36a cat 
/proc/sys/net/ipv6/conf/qr-7fad6b1b-c9/accept_ra

1
[root@controller2 ~]# ip netns exec 
qrouter-0775785e-a93a-4501-917b-be92ff03f36a cat 
/proc/sys/net/ipv6/conf/qr-5be04815-68/accept_ra

1

Perhaps the accept_ra patches does not apply for enable/disable or 
routers changing from a normal router to a L3 HA router?

Best regards

On 08/20/2018 11:50 AM, Tobias Urdin wrote:
Ok, so the issue here seems to be that I have a L3 HA router with 
SLAAC, both the active and standby router will
configure the SLAAC obtained address causing a conflict since both 
side share the same MAC address.


Is there any workaround for this? Should SLAAC even be enabled for 
interfaces on the standby router?


Best regards
Tobias

On 08/20/2018 11:37 AM, Tobias Urdin wrote:

Forgot [neutron] tag.

On 08/20/2018 11:36 AM, Tobias Urdin wrote:

Hello,

Note: before reading, this router was a regular router but was then 
disable, changed ha=true so it's now a L3 HA router, then it was 
enabled again.

CC openstack-dev for help or feedback if it's a possible bug.

I've been testing around with IPv6 and overall the experience has 
been positive but I've met some weird issue that I cannot put my 
head around.
So this is a neutron L3 router with an outside interface with a ipv4 
and ipv6 from the provider network and one inside interface for ipv4 
and one inside interface for ipv6.


The instances for some reason get's there default gateway as the 
ipv6 link-local (in fe80::/10) from the router with SLAAC and radvd.


(. is provider network, . is inside network, they 
are masked so don't pay attention to the number per se)


*interfaces inside router:*
15: ha-9bde1bb1-bd:  mtu 1450 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:05:80:32 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.7/18 brd 169.254.255.255 scope global 
ha-9bde1bb1-bd

   valid_lft forever preferred_lft forever
    inet 169.254.0.1/24 scope global ha-9bde1bb1-bd
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe05:8032/64 scope link
   valid_lft forever preferred_lft forever
19: qg-86e465f6-33:  mtu 1500 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:3b:8b:a5 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/22 scope global qg-86e465f6-33
   valid_lft forever preferred_lft forever
    inet6 :::f/64 scope global nodad
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe3b:8ba5/64 scope link nodad
   valid_lft forever preferred_lft forever
1168: qr-5be04815-68:  mtu 1450 
qdisc noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:c3:85:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.1/24 scope global qr-5be04815-68
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec3:85bd/64 scope link
   valid_lft forever preferred_lft forever
1169: qr-7fad6b1b-c9:  mtu 1450 
qdisc noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:66:de:a8 brd ff:ff:ff:ff:ff:ff
    inet6 ::0:1::1/64 scope global nodad
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe66:dea8/64 scope link
   valid_lft forever preferred_lft forever

I get this error messages in dmesg on the network node:
[581085.858869] IPv6: qr-5be04815-68: IPv6 duplicate address 
::0:1:f816:3eff:fec3:85bd detected!
[581085.997497] IPv6: qr-7fad6b1b-c9: IPv6 duplicate address 
::0:1:f816:3eff:fe66:dea8 detected!
[581142.869939] IPv6: qr-5be04815-68: IPv6 duplicate address 
::0:1:f816:3eff:fec3:85bd detected!
[581143.182371] IPv6: qr-7fad6b1b-c9: IPv6 duplicate address 
::0:1:f816:3eff:fe66:dea8 detected!


*radvd:*
interface qr-7fad6b1b-c9
{
   AdvSendAdvert on;
   MinRtrAdvInterval 30;
   MaxRtrAdvInterval 100;

   AdvLinkMTU 1450;

   RDNSS  2001:4860:4860::  {};

   prefix ::0:1::/64
   {
    AdvOnLink on;
    AdvAutonomous on;
   };
};

*inside instance:*
ipv4 = 192.168.199.7
ipv6 = ::0:1:f816:3eff:fe29:723d/64 (from radvd SLAAC)

I can ping ipv4 gateway 192.168.199.1 and internet over ipv4.
I can ping ipv6 gateway ::0:1::1 but I can't ping the internet

checking the ipv6 routing table on my instance I either get no 
default gateway at all or I get a default gateway to a fe80::/10 
link-local address.

IIRC this worked before I changed the router to a L3 HA router.

Appreciate any feedback!

Best regards
Tobias






__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http

Re: [openstack-dev] [neutron] neutron ipv6 radvd sends out link-local or nothing as def gw (L3 HA issue?)

[Tobias Urdin]

When I removed those ips and set accept_ra to 0 on the backup router:

ip netns exec qrouter-0775785e-a93a-4501-917b-be92ff03f36a sysctl -w 
net.ipv6.conf.qr-7fad6b1b-c9.accept_ra=0
ip netns exec qrouter-0775785e-a93a-4501-917b-be92ff03f36a sysctl -w 
net.ipv6.conf.qr-5be04815-68.accept_ra=0

ip netns exec qrouter-0775785e-a93a-4501-917b-be92ff03f36a ip a l
ip netns exec qrouter-0775785e-a93a-4501-917b-be92ff03f36a ip addr del 
::0:1:f816:3eff:fe66:dea8/64 dev qr-7fad6b1b-c9
ip netns exec qrouter-0775785e-a93a-4501-917b-be92ff03f36a ip addr del 
::0:1:f816:3eff:fec3:85bd/64 dev qr-5be04815-68


And enabled ipv6 forwarding on the active router:
ip netns exec qrouter-0775785e-a93a-4501-917b-be92ff03f36a sysctl -w 
net.ipv6.conf.all.forwarding=1


It started working again, I think this is an issue when disabling a 
router, change it to L3 HA and enable it again, so a bug?


Best regards
Tobias

On 08/20/2018 11:58 AM, Tobias Urdin wrote:
Continuing forward, these patches should've fixed that 
https://review.openstack.org/#/q/topic:bug/1667756+(status:open+OR+status:merged)

I'm on Queens.

The two inside interfaces on the backup router:
[root@controller2 ~]# ip netns exec 
qrouter-0775785e-a93a-4501-917b-be92ff03f36a cat 
/proc/sys/net/ipv6/conf/qr-7fad6b1b-c9/accept_ra

1
[root@controller2 ~]# ip netns exec 
qrouter-0775785e-a93a-4501-917b-be92ff03f36a cat 
/proc/sys/net/ipv6/conf/qr-5be04815-68/accept_ra

1

Perhaps the accept_ra patches does not apply for enable/disable or 
routers changing from a normal router to a L3 HA router?

Best regards

On 08/20/2018 11:50 AM, Tobias Urdin wrote:
Ok, so the issue here seems to be that I have a L3 HA router with 
SLAAC, both the active and standby router will
configure the SLAAC obtained address causing a conflict since both 
side share the same MAC address.


Is there any workaround for this? Should SLAAC even be enabled for 
interfaces on the standby router?


Best regards
Tobias

On 08/20/2018 11:37 AM, Tobias Urdin wrote:

Forgot [neutron] tag.

On 08/20/2018 11:36 AM, Tobias Urdin wrote:

Hello,

Note: before reading, this router was a regular router but was then 
disable, changed ha=true so it's now a L3 HA router, then it was 
enabled again.

CC openstack-dev for help or feedback if it's a possible bug.

I've been testing around with IPv6 and overall the experience has 
been positive but I've met some weird issue that I cannot put my 
head around.
So this is a neutron L3 router with an outside interface with a 
ipv4 and ipv6 from the provider network and one inside interface 
for ipv4 and one inside interface for ipv6.


The instances for some reason get's there default gateway as the 
ipv6 link-local (in fe80::/10) from the router with SLAAC and radvd.


(. is provider network, . is inside network, they 
are masked so don't pay attention to the number per se)


*interfaces inside router:*
15: ha-9bde1bb1-bd:  mtu 1450 
qdisc noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:05:80:32 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.7/18 brd 169.254.255.255 scope global 
ha-9bde1bb1-bd

   valid_lft forever preferred_lft forever
    inet 169.254.0.1/24 scope global ha-9bde1bb1-bd
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe05:8032/64 scope link
   valid_lft forever preferred_lft forever
19: qg-86e465f6-33:  mtu 1500 
qdisc noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:3b:8b:a5 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/22 scope global qg-86e465f6-33
   valid_lft forever preferred_lft forever
    inet6 :::f/64 scope global nodad
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe3b:8ba5/64 scope link nodad
   valid_lft forever preferred_lft forever
1168: qr-5be04815-68:  mtu 1450 
qdisc noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:c3:85:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.1/24 scope global qr-5be04815-68
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec3:85bd/64 scope link
   valid_lft forever preferred_lft forever
1169: qr-7fad6b1b-c9:  mtu 1450 
qdisc noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:66:de:a8 brd ff:ff:ff:ff:ff:ff
    inet6 ::0:1::1/64 scope global nodad
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe66:dea8/64 scope link
   valid_lft forever preferred_lft forever

I get this error messages in dmesg on the network node:
[581085.858869] IPv6: qr-5be04815-68: IPv6 duplicate address 
::0:1:f816:3eff:fec3:85bd detected!
[581085.997497] IPv6: qr-7fad6b1b-c9: IPv6 duplicate address 
::0:1:f816:3eff:fe66:dea8 detected!
[581142.869939] IPv6: qr-5be04815-68: IPv6 duplicate address 
::0:1:f816:3eff:fec3:85bd detected!
[581143.182371] IPv6: qr-7fad6b1b-c9: IPv6 duplicate address 
::0:1:f816:3eff:fe66:dea8 detected!


*radvd:*
interface qr-7fad6b1b-c9

[openstack-dev] neutron ipv6 radvd sends out link-local or nothing as def gw (L3 HA issue?)

[Tobias Urdin]

Hello,

Note: before reading, this router was a regular router but was then 
disable, changed ha=true so it's now a L3 HA router, then it was enabled 
again.

CC openstack-dev for help or feedback if it's a possible bug.

I've been testing around with IPv6 and overall the experience has been 
positive but I've met some weird issue that I cannot put my head around.
So this is a neutron L3 router with an outside interface with a ipv4 and 
ipv6 from the provider network and one inside interface for ipv4 and one 
inside interface for ipv6.


The instances for some reason get's there default gateway as the ipv6 
link-local (in fe80::/10) from the router with SLAAC and radvd.


(. is provider network, . is inside network, they are 
masked so don't pay attention to the number per se)


*interfaces inside router:*
15: ha-9bde1bb1-bd:  mtu 1450 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:05:80:32 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.7/18 brd 169.254.255.255 scope global ha-9bde1bb1-bd
   valid_lft forever preferred_lft forever
    inet 169.254.0.1/24 scope global ha-9bde1bb1-bd
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe05:8032/64 scope link
   valid_lft forever preferred_lft forever
19: qg-86e465f6-33:  mtu 1500 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:3b:8b:a5 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/22 scope global qg-86e465f6-33
   valid_lft forever preferred_lft forever
    inet6 :::f/64 scope global nodad
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe3b:8ba5/64 scope link nodad
   valid_lft forever preferred_lft forever
1168: qr-5be04815-68:  mtu 1450 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:c3:85:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.1/24 scope global qr-5be04815-68
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec3:85bd/64 scope link
   valid_lft forever preferred_lft forever
1169: qr-7fad6b1b-c9:  mtu 1450 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:66:de:a8 brd ff:ff:ff:ff:ff:ff
    inet6 ::0:1::1/64 scope global nodad
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe66:dea8/64 scope link
   valid_lft forever preferred_lft forever

I get this error messages in dmesg on the network node:
[581085.858869] IPv6: qr-5be04815-68: IPv6 duplicate address 
::0:1:f816:3eff:fec3:85bd detected!
[581085.997497] IPv6: qr-7fad6b1b-c9: IPv6 duplicate address 
::0:1:f816:3eff:fe66:dea8 detected!
[581142.869939] IPv6: qr-5be04815-68: IPv6 duplicate address 
::0:1:f816:3eff:fec3:85bd detected!
[581143.182371] IPv6: qr-7fad6b1b-c9: IPv6 duplicate address 
::0:1:f816:3eff:fe66:dea8 detected!


*radvd:*
interface qr-7fad6b1b-c9
{
   AdvSendAdvert on;
   MinRtrAdvInterval 30;
   MaxRtrAdvInterval 100;

   AdvLinkMTU 1450;

   RDNSS  2001:4860:4860::  {};

   prefix ::0:1::/64
   {
    AdvOnLink on;
    AdvAutonomous on;
   };
};

*inside instance:*
ipv4 = 192.168.199.7
ipv6 = ::0:1:f816:3eff:fe29:723d/64 (from radvd SLAAC)

I can ping ipv4 gateway 192.168.199.1 and internet over ipv4.
I can ping ipv6 gateway ::0:1::1 but I can't ping the internet

checking the ipv6 routing table on my instance I either get no default 
gateway at all or I get a default gateway to a fe80::/10 link-local address.

IIRC this worked before I changed the router to a L3 HA router.

Appreciate any feedback!

Best regards
Tobias
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [neutron] neutron ipv6 radvd sends out link-local or nothing as def gw (L3 HA issue?)

[Tobias Urdin]

Forgot [neutron] tag.

On 08/20/2018 11:36 AM, Tobias Urdin wrote:

Hello,

Note: before reading, this router was a regular router but was then 
disable, changed ha=true so it's now a L3 HA router, then it was 
enabled again.

CC openstack-dev for help or feedback if it's a possible bug.

I've been testing around with IPv6 and overall the experience has been 
positive but I've met some weird issue that I cannot put my head around.
So this is a neutron L3 router with an outside interface with a ipv4 
and ipv6 from the provider network and one inside interface for ipv4 
and one inside interface for ipv6.


The instances for some reason get's there default gateway as the ipv6 
link-local (in fe80::/10) from the router with SLAAC and radvd.


(. is provider network, . is inside network, they are 
masked so don't pay attention to the number per se)


*interfaces inside router:*
15: ha-9bde1bb1-bd:  mtu 1450 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:05:80:32 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.7/18 brd 169.254.255.255 scope global ha-9bde1bb1-bd
   valid_lft forever preferred_lft forever
    inet 169.254.0.1/24 scope global ha-9bde1bb1-bd
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe05:8032/64 scope link
   valid_lft forever preferred_lft forever
19: qg-86e465f6-33:  mtu 1500 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:3b:8b:a5 brd ff:ff:ff:ff:ff:ff
    inet 1.2.3.4/22 scope global qg-86e465f6-33
   valid_lft forever preferred_lft forever
    inet6 :::f/64 scope global nodad
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe3b:8ba5/64 scope link nodad
   valid_lft forever preferred_lft forever
1168: qr-5be04815-68:  mtu 1450 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:c3:85:bd brd ff:ff:ff:ff:ff:ff
    inet 192.168.99.1/24 scope global qr-5be04815-68
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec3:85bd/64 scope link
   valid_lft forever preferred_lft forever
1169: qr-7fad6b1b-c9:  mtu 1450 qdisc 
noqueue state UNKNOWN group default qlen 1000

    link/ether fa:16:3e:66:de:a8 brd ff:ff:ff:ff:ff:ff
    inet6 ::0:1::1/64 scope global nodad
   valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe66:dea8/64 scope link
   valid_lft forever preferred_lft forever

I get this error messages in dmesg on the network node:
[581085.858869] IPv6: qr-5be04815-68: IPv6 duplicate address 
::0:1:f816:3eff:fec3:85bd detected!
[581085.997497] IPv6: qr-7fad6b1b-c9: IPv6 duplicate address 
::0:1:f816:3eff:fe66:dea8 detected!
[581142.869939] IPv6: qr-5be04815-68: IPv6 duplicate address 
::0:1:f816:3eff:fec3:85bd detected!
[581143.182371] IPv6: qr-7fad6b1b-c9: IPv6 duplicate address 
::0:1:f816:3eff:fe66:dea8 detected!


*radvd:*
interface qr-7fad6b1b-c9
{
   AdvSendAdvert on;
   MinRtrAdvInterval 30;
   MaxRtrAdvInterval 100;

   AdvLinkMTU 1450;

   RDNSS  2001:4860:4860::  {};

   prefix ::0:1::/64
   {
    AdvOnLink on;
    AdvAutonomous on;
   };
};

*inside instance:*
ipv4 = 192.168.199.7
ipv6 = ::0:1:f816:3eff:fe29:723d/64 (from radvd SLAAC)

I can ping ipv4 gateway 192.168.199.1 and internet over ipv4.
I can ping ipv6 gateway ::0:1::1 but I can't ping the internet

checking the ipv6 routing table on my instance I either get no default 
gateway at all or I get a default gateway to a fe80::/10 link-local 
address.

IIRC this worked before I changed the router to a L3 HA router.

Appreciate any feedback!

Best regards
Tobias


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [magnum] [magnum-ui] show certificate button bug requesting reviews

[Tobias Urdin]

Hello,

Requesting reviews from the magnum-ui core team for 
https://review.openstack.org/#/c/595245/
I'm hoping that we could make quick due of this and be able to backport 
it to the stable/rocky release, would be ideal to backport it for 
stable/queens as well.


Best regards
Tobias

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [magnum] supported OS images and magnum spawn failures for Swarm and Kubernetes

[Tobias Urdin]

Thanks for all of your help everyone,

I've been busy with other thing but was able to pick up where I left 
regarding Magnum.
After fixing some issues I have been able to provision a working 
Kubernetes cluster.


I'm still having issues with getting Docker Swarm working, I've tried 
with both Docker and flannel as the networking layer but
none of these works. After investigating the issue seems to be that 
etcd.service is not installed (unit file doesn't exist) so the master
doesn't work, the minion swarm node is provisioned but cannot join the 
cluster because there is no etcd.


Anybody seen this issue before? I've been digging through all cloud-init 
logs and cannot see anything that would cause this.


I also have another separate issue, when provisioning using the 
magnum-ui in Horizon and selecting ubuntu with Mesos I get the error
"The Parameter (nodes_affinity_policy) was not provided". The 
nodes_affinity_policy do have a default value in magnum.conf so I'm starting

to think this might be an issue with the magnum-ui dashboard?

Best regards
Tobias

On 08/04/2018 06:24 PM, Joe Topjian wrote:
We recently deployed Magnum and I've been making my way through 
getting both Swarm and Kubernetes running. I also ran into some 
initial issues. These notes may or may not help, but thought I'd share 
them in case:


* We're using Barbican for SSL. I have not tried with the internal 
x509keypair.


* I was only able to get things running with Fedora Atomic 27, 
specifically the version used in the Magnum docs: 
https://docs.openstack.org/magnum/latest/install/launch-instance.html


Anything beyond that wouldn't even boot in my cloud. I haven't dug 
into this.


* Kubernetes requires a Cluster Template to have a label of 
cert_manager_api=true set in order for the cluster to fully come up 
(at least, it didn't work for me until I set this).


As far as troubleshooting methods go, check the cloud-init logs on the 
individual instances to see if any of the "parts" have failed to run. 
Manually re-run the parts on the command-line to get a better idea of 
why they failed. Review the actual script, figure out the variable 
interpolation and how it relates to the Cluster Template being used.


Eventually I was able to get clusters running with the stock 
driver/templates, but wanted to tune them in order to better fit in 
our cloud, so I've "forked" them. This is in no way a slight against 
the existing drivers/templates nor do I recommend doing this until you 
reach a point where the stock drivers won't meet your needs. But I 
mention it because it's possible to do and it's not terribly hard. 
This is still a work-in-progress and a bit hacky:


https://github.com/cybera/magnum-templates

Hope that helps,
Joe

On Fri, Aug 3, 2018 at 6:46 AM, Tobias Urdin <mailto:tobias.ur...@binero.se>> wrote:


Hello,

I'm testing around with Magnum and have so far only had issues.
I've tried deploying Docker Swarm (on Fedora Atomic 27, Fedora
Atomic 28) and Kubernetes (on Fedora Atomic 27) and haven't been
able to get it working.

Running Queens, is there any information about supported images?
Is Magnum maintained to support Fedora Atomic still?
What is in charge of population the certificates inside the
instances, because this seems to be the root of all issues, I'm
not using Barbican but the x509keypair driver
is that the reason?

Perhaps I missed some documentation that x509keypair does not
support what I'm trying to do?

I've seen the following issues:

Docker:
* Master does not start and listen on TCP because of certificate
issues
dockerd-current[1909]: Could not load X509 key pair (cert:
"/etc/docker/server.crt", key: "/etc/docker/server.key")

* Node does not start with:
Dependency failed for Docker Application Container Engine.
docker.service: Job docker.service/start failed with result
'dependency'.

Kubernetes:
* Master etcd does not start because /run/etcd does not exist
** When that is created it fails to start because of certificate
2018-08-03 12:41:16.554257 C | etcdmain: open
/etc/etcd/certs/server.crt: no such file or directory

* Master kube-apiserver does not start because of certificate
unable to load server certificate: open
/etc/kubernetes/certs/server.crt: no such file or directory

* Master heat script just sleeps forever waiting for port 8080 to
become available (kube-apiserver) so it can never kubectl apply
the final steps.

* Node does not even start and times out when Heat deploys it,
probably because master never finishes

Any help is appreciated perhaps I've missed something crucial,
I've not tested Kubernetes on CoreOS yet.

Best regards
Tobias

__
OpenStack Development Mailing List (not f

[openstack-dev] [puppet-openstack] [announce] puppet-openstack now has Ubuntu 18.04 Bionic support

[Tobias Urdin]

Hello Stackers,

Would just like to give a heads-up that the puppet-openstack project as
of the Rocky release will supports
Ubuntu 18.04 Bionic and we are as of yesterday/today checking that in
infra Zuul CI.

As a step for adding this support we also introduced support for the
Ceph Mimic release for the puppet-ceph module.
Because of upstream packaging Ceph Mimic cannot be used on Debian 9, and
should also note that Ceph Luminous cannot be
used on Ubuntu 18.04 Bionic using upstream Ceph community packages
(Canonical is packaging Ceph in Bionic main repo).

I would like to thank everybody contributing to this effort and for
everyone involved in the puppet-openstack project that has reviewed all
changes.
A special thanks to all the infra-people that has helped out a bunch
with mirrors, Zuul and providing all necessary bits required to work on
this.

Best regards
Tobias

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [election] [tc] thank you

[Tobias Urdin]

Emilien,

You've been an inspiration to the community and to me personally!
Thanks for helping making OpenStack better in all aspects.

Best regards
Tobias

On 09/05/2018 04:15 AM, Emilien Macchi wrote:
After 2 years at the TC, I feel lucky enough to have been part of this 
group where I hope that my modest contributions helped to make 
OpenStack a bit better.
I've learnt so many things and worked with a talented group where it's 
not easy every day, but we have made and will continue to progress in 
the future.
I personally feel like some rotation needs to happen, therefore I 
won't run the current election.


I don't plan to leave or anything, I just wanted to say "merci" to the 
community who gave me a chance to be part of this team.

--
Emilien Macchi


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] Puppet weekly recap - week 35-36

[Tobias Urdin]

Hello fellow Puppeteers!

Welcome to the weekly Puppet recap for week 35 and 36.
Because I was away last week I forgot to follow up the progress of week 35.

The past two weeks has been quite calm, we have had a lot of changes due 
to the move
away from Zuul config in project-config and the bump of the version for 
all modules on the

master branch to prepare for the Stein cycle.

MERGED CHANGES


= puppet-barbican =
* 34e6f3a Add barbican::worker class
The Barbican module now supports installation and management of the
barbican-worker service.

= puppet-cinder =
* 3c634d2 Deprecate parameters that have been removed from cinder
Config options for Cinder that was removed in Queens has been deprecated
in the Puppet interface.

= puppet-ironic =
* f37d8f6 Add tests for oslo_messaging_notifications
Added missing spec tests for the recent oslo messaging addition

* 27bf3a0 Expose the notification_level paramenter
Added the [DEFAULT]/notification_level configuration option

= puppet-neutron =
* 33f8cdc Notify about deprecated neutron::services::lbaas
Neutron LBaaS is deprecated so a warning has been added

* c70e4fa Add ensure_lbaas_package to neutron::server
Added the ability to install the LBaaS plugin from the neutron::server class

= puppet-nova =
393694a Add a release note for sync_power_state_interval parameter
b4f3d6a compute: add sync_power_state_interval parameter

Added the sync_power_state_interval configuration option for nova.

= puppet-octavia =
2b83ae2 Add octavia::certificates::client_ca and data
45673ee Added missing DB params to init class
e1531c3 Add Octavia API WSGI support
d2a9586 Add octavia::neutron to configure nova section
6731e53 Add octavia::glance to configure glance section
9b285e7 Add missing options to octavia::certificates
6864cd0 Add octavia::nova to configure nova section
7d6bada Add workers support to octavia::worker class
6e7dacc Add api_v1_enabled and api_v2_enabled options
14c5257 Add allow_tls_terminated_listeners config option

The Octavia module have had a lot of changes to make sure it's fit
to be used. It now includes all the required classes and configuration
options to use it. You can run the API in WSGI, enable/disable the v1
and v2 API, set if TLS listeners is allowed and separate the client and
server CA certificates.

= puppet-openstack-integration =
* 1edb135 Remove tripleo-puppet-ci-centos-7-undercloud-oooq job
Removed TripleO testing for non-containerized undercloud.

* 9cb0e06 Higher timeout and two tries for update packages
In the repos.pp file the upgrade packages call times out so we tried
increasing the timeout and set tries to two (2) but it did not solve the 
issue.


* 15dd562 Add barbican::worker to integration testing
The newly added barbican::worker class is tested in the integration testing.

= puppet-vswitch =
* b6dab2e Fix the undefined method 'chomp' for nil:NilClass error seen 
with ovs 2.10
Fixed a bug where the output to stdout contained error messages the 
provider would

fail to parse the values, it now ignores nil values.

SPECS
=
No new specs.

Only one open spec:

*  Add parameter data types spec
https://review.openstack.org/#/c/568929/

CI
=

There is some current issues with the CI, if anybody feels they have 
time we all would appreciate

that you help us resolve it.

* The update-packages call in repos.pp for 
openstack/puppet-openstack-integration times out on
CentOS 7 (calling yum upgrade). This makes integration testing for 
puppet-openstacklib fail.


* The stable/ocata and stable/pike branches has issues and are failing, 
this block most of the
Zuul config retire from project-config patches that Doug has proposed, 
we need to resolve this by
unblocking (fixing) the CI. This is probably backporting previous fixes 
to CI.


I have been very busy with finalizing a project so I've not been able to 
look at this. I will have a look

this weekend or hopefully next week but would appreciate any help.

After CI is fixed we can merge all these:

* Update Gemfile for stable/rocky (openstack/puppet-openstacklib)
https://review.openstack.org/#/c/597087/

* make openstackclient package name configurable 
(openstack/puppet-openstacklib)

https://review.openstack.org/#/c/599015/

* All the python3-first topic changes
(named "import zuul job settings from project-config" and "switch 
documentation job to new PTI")


NEEDS REVIEWS
==

* Make ironic password a secret
https://review.openstack.org/#/c/598173/

* Remove usage of deprecated RamFilter
https://review.openstack.org/#/c/597204/

* Add cinder::nova class to configure nova section
https://review.openstack.org/#/c/600559/

* Cinder-type param is_public/access_project_ids
https://review.openstack.org/#/c/600071/

* Remove ironic inspector dnsmasq bind-interfaces setting
https://review.openstack.org/#/c/600068/

* Add octavia testing
https://review.openstack.org/#/c/597600/

* Change beaker testing to 

Re: [openstack-dev] [goals][python3][adjutant][barbican][chef][cinder][cloudkitty][i18n][infra][loci][nova][charms][rpm][puppet][qa][telemetry][trove] join the bandwagon!

[Tobias Urdin]

Hello Doug,

I've proposed moving all job config from project-config to the repos [1].
I don't know what to do with the periodic job here [2] should that be 
left in project-config or moved?


Best regards
Tobias

[1] https://review.openstack.org/#/q/topic:move-zuul-config
[2] 
https://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L10891


On 08/31/2018 01:34 AM, Doug Hellmann wrote:

Below is the list of project teams that have not yet started migrating
their zuul configuration. If you're ready to go, please respond to this
email to let us know so we can start proposing patches.

Doug

| adjutant| 3 repos   |
| barbican| 5 repos   |
| Chef OpenStack  | 19 repos  |
| cinder  | 6 repos   |
| cloudkitty  | 5 repos   |
| I18n| 2 repos   |
| Infrastructure  | 158 repos |
| loci| 1 repos   |
| nova| 6 repos   |
| OpenStack Charms| 80 repos  |
| Packaging-rpm   | 4 repos   |
| Puppet OpenStack| 47 repos  |
| Quality Assurance   | 22 repos  |
| Telemetry   | 8 repos   |
| trove   | 5 repos   |

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [goals][python3][adjutant][barbican][chef][cinder][cloudkitty][i18n][infra][loci][nova][charms][rpm][puppet][qa][telemetry][trove] join the bandwagon!

[Tobias Urdin]

Oh, that's bad. I will abandon.
It's a go for Puppet then.

Best regards

On 08/31/2018 12:36 PM, Andreas Jaeger wrote:

On 2018-08-31 10:49, Tobias Urdin wrote:

Hello Doug,

I've proposed moving all job config from project-config to the repos [1].
I don't know what to do with the periodic job here [2] should that be
left in project-config or moved?


Tobias, I'm sorry to see you doing this - but please abandon!  You've
done only part of the work and finishing it will complicate it. Doug has
scripts for these and can easily run that. What Doug asked for was for
an official "go ahead" from the puppet team,

Andreas



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] puppet-senlin development

[Tobias Urdin]

Hello Alex,

I personally don't know about any entity specifically working on the Puppet 
Senlin module.

We strongly welcome anybody to contribute to the development of the Puppet 
OpenStack modules.

We are happy to help :)

Best regards
Tobias

Sent from my iPhone

On 9 Jul 2018, at 16:00, Alexandru Sorodoc 
mailto:a...@privacysystems.eu>> wrote:


Hello,

Is anyone working or planning to work on the puppet-senlin module? We want to 
use Senlin in our Pike deployment and we are considering contributing to its 
puppet module to bring it to a working state.

Best regards,
Alex

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [nova] Default scheduler filters survey

[Tobias Urdin]

Two different setups, very basic.

AggregateInstanceExtraSpecsFilter
RetryFilter
AvailabilityZoneFilter
ComputeFilter
ComputeCapabilitiesFilter
ImagePropertiesFilter
ServerGroupAntiAffinityFilter
ServerGroupAffinityFilter

RetryFilter
AvailabilityZoneFilter
RamFilter
ComputeFilter
ComputeCapabilitiesFilter
ImagePropertiesFilter
ServerGroupAntiAffinityFilter
ServerGroupAffinityFilter

On 04/18/2018 06:34 PM, Chris Friesen wrote:
> On 04/18/2018 09:17 AM, Artom Lifshitz wrote:
>
>> To that end, we'd like to know what filters operators are enabling in
>> their deployment. If you can, please reply to this email with your
>> [filter_scheduler]/enabled_filters (or
>> [DEFAULT]/scheduler_default_filters if you're using an older version)
>> option from nova.conf. Any other comments are welcome as well :)
> RetryFilter
> ComputeFilter
> AvailabilityZoneFilter
> AggregateInstanceExtraSpecsFilter
> ComputeCapabilitiesFilter
> ImagePropertiesFilter
> NUMATopologyFilter
> ServerGroupAffinityFilter
> ServerGroupAntiAffinityFilter
> PciPassthroughFilter
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Octavia] SSL errors polling amphorae and missing tenant network interface

[Tobias Urdin]

Hello,

I've been having a lot of issues with SSL certificates myself, on my 
second trip now trying to get it working.


Before I spent a lot of time walking through every line in the DevStack 
plugin and fixing my config options, used the generate

script [1] and still it didn't work.

When I got the "invalid padding" issue it was because of the DN I used 
for the CA and the certificate IIRC.


> 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING 
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect 
to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa 
routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'), 
('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'), 
('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
> 19:47 < tobias-urdin> after a quick google "The problem was that my 
CA DN was the same as the certificate DN."


IIRC I think that solved it, but then again I wouldn't remember fully 
since I've been at so many different angles by now.


Here is my IRC logs history from the #openstack-lbaas channel, perhaps 
it can help you out

http://paste.openstack.org/show/732575/

-

Sorry for hijacking the thread but I'm stuck as well.

I've in the past tried to generate the certificates with [1] but now 
moved on to using the openstack-ansible way of generating them [2]

with some modifications.

Right now I'm just getting: Could not connect to instance. Retrying.: 
SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
from the amphoras, haven't got any further but I've eliminated a lot of 
stuck in the middle.


Tried deploying Ocatavia on Ubuntu with python3 to just make sure there 
wasn't an issue with CentOS and OpenSSL versions since it tends to lag 
behind.
Checking the amphora with openssl s_client [3] it gives the same one, 
but the verification is successful just that I don't understand what the 
bad signature
part is about, from browsing some OpenSSL code it seems to be related to 
RSA signatures somehow.


140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad 
signature:s3_clnt.c:2032:


So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS 
(openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm 
back to something related
to the certificates or the communication between the endpoints, or what 
actually responds inside the amphora (gunicorn IIUC?). Based on the
"verify" functions actually causing that bad signature error I would 
assume it's the generated certificate that the amphora presents that is 
causing it.


I'll have to continue the troubleshooting to the inside of the amphora, 
I've used the test-only amphora image before but have now built my own 
one that is
using the amphora-agent from the actual stable branch, but same issue 
(bad signature).


For verbosity this is the config options set for the certificates in 
octavia.conf and which file it was copied from [4], same here, a 
replication of what openstack-ansible does.


Appreciate any feedback or help :)

Best regards
Tobias

[1] 
https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh

[2] http://paste.openstack.org/show/732483/
[3] http://paste.openstack.org/show/732486/
[4] http://paste.openstack.org/show/732487/

On 10/20/2018 01:53 AM, Michael Johnson wrote:

Hi Erik,

Sorry to hear you are still having certificate issues.

Issue #2 is probably caused by issue #1. Since we hot-plug the tenant
network for the VIP, one of the first steps after the worker connects
to the amphora agent is finishing the required configuration of the
VIP interface inside the network namespace on the amphroa.

If I remember correctly, you are attempting to configure Octavia with
the dual CA option (which is good for non-development use).

This is what I have for notes:

[certificates] gets the following:
cert_generator = local_cert_generator
ca_certificate = server CA's "server.pem" file
ca_private_key = server CA's "server.key" file
ca_private_key_passphrase = pass phrase for ca_private_key
  [controller_worker]
  client_ca = Client CA's ca_cert file
  [haproxy_amphora]
client_cert = Client CA's client.pem file (I think with it's key
concatenated is what rm_work said the other day)
server_ca = Server CA's ca_cert file

That said, I can probably run through this and write something up next
week that is more step-by-step/detailed.

Michael

On Fri, Oct 19, 2018 at 2:31 PM Erik McCormick
 wrote:

Apologies for cross-posting, but in the event that these might be
worth filing as bugs, I wanted the Octavia devs to see it as well...

I've been wrestling with getting Octavia up and running and have
become stuck on two issues. I'm hoping someone has run into these
before. My google foo has come up empty.

Issue 1:
When the Octavia controller tries to poll the amphora instance, it
tries repeatedly and eventually

Re: [openstack-dev] [Octavia] SSL errors polling amphorae and missing tenant network interface

[Tobias Urdin]

+operators

My bad.

On 10/22/2018 10:22 AM, Tobias Urdin wrote:

Hello,

I've been having a lot of issues with SSL certificates myself, on my
second trip now trying to get it working.

Before I spent a lot of time walking through every line in the DevStack
plugin and fixing my config options, used the generate
script [1] and still it didn't work.

When I got the "invalid padding" issue it was because of the DN I used
for the CA and the certificate IIRC.

  > 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
  > 19:47 < tobias-urdin> after a quick google "The problem was that my
CA DN was the same as the certificate DN."

IIRC I think that solved it, but then again I wouldn't remember fully
since I've been at so many different angles by now.

Here is my IRC logs history from the #openstack-lbaas channel, perhaps
it can help you out
http://paste.openstack.org/show/732575/

-

Sorry for hijacking the thread but I'm stuck as well.

I've in the past tried to generate the certificates with [1] but now
moved on to using the openstack-ansible way of generating them [2]
with some modifications.

Right now I'm just getting: Could not connect to instance. Retrying.:
SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
from the amphoras, haven't got any further but I've eliminated a lot of
stuck in the middle.

Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
wasn't an issue with CentOS and OpenSSL versions since it tends to lag
behind.
Checking the amphora with openssl s_client [3] it gives the same one,
but the verification is successful just that I don't understand what the
bad signature
part is about, from browsing some OpenSSL code it seems to be related to
RSA signatures somehow.

140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
signature:s3_clnt.c:2032:

So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS
(openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm
back to something related
to the certificates or the communication between the endpoints, or what
actually responds inside the amphora (gunicorn IIUC?). Based on the
"verify" functions actually causing that bad signature error I would
assume it's the generated certificate that the amphora presents that is
causing it.

I'll have to continue the troubleshooting to the inside of the amphora,
I've used the test-only amphora image before but have now built my own
one that is
using the amphora-agent from the actual stable branch, but same issue
(bad signature).

For verbosity this is the config options set for the certificates in
octavia.conf and which file it was copied from [4], same here, a
replication of what openstack-ansible does.

Appreciate any feedback or help :)

Best regards
Tobias

[1]
https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh
[2] http://paste.openstack.org/show/732483/
[3] http://paste.openstack.org/show/732486/
[4] http://paste.openstack.org/show/732487/

On 10/20/2018 01:53 AM, Michael Johnson wrote:

Hi Erik,

Sorry to hear you are still having certificate issues.

Issue #2 is probably caused by issue #1. Since we hot-plug the tenant
network for the VIP, one of the first steps after the worker connects
to the amphora agent is finishing the required configuration of the
VIP interface inside the network namespace on the amphroa.

If I remember correctly, you are attempting to configure Octavia with
the dual CA option (which is good for non-development use).

This is what I have for notes:

[certificates] gets the following:
cert_generator = local_cert_generator
ca_certificate = server CA's "server.pem" file
ca_private_key = server CA's "server.key" file
ca_private_key_passphrase = pass phrase for ca_private_key
   [controller_worker]
   client_ca = Client CA's ca_cert file
   [haproxy_amphora]
client_cert = Client CA's client.pem file (I think with it's key
concatenated is what rm_work said the other day)
server_ca = Server CA's ca_cert file

That said, I can probably run through this and write something up next
week that is more step-by-step/detailed.

Michael

On Fri, Oct 19, 2018 at 2:31 PM Erik McCormick
 wrote:

Apologies for cross-posting, but in the event that these might be
worth filing as bugs, I wanted the Octavia devs to see it as well...

I've been wrestling with getting Octavia up and running and have
become stuck on two issues. I'm hoping someone has run into these
before. My google foo has come up empty.

Issue 1:
When the Octavia controller tries to poll the amphora instance, it
tries repeatedly and even

Re: [openstack-dev] [Octavia] [Kolla] SSL errors polling amphorae and missing tenant network interface

[Tobias Urdin]

Hello Erik,

Could you specify the DNs you used for all certificates just so that I 
can rule it out on my side.
You can redact anything sensitive with some to just get the feel on how 
it's configured.


Best regards
Tobias

On 10/22/2018 04:47 PM, Erik McCormick wrote:

On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin  wrote:

Hello,

I've been having a lot of issues with SSL certificates myself, on my
second trip now trying to get it working.

Before I spent a lot of time walking through every line in the DevStack
plugin and fixing my config options, used the generate
script [1] and still it didn't work.

When I got the "invalid padding" issue it was because of the DN I used
for the CA and the certificate IIRC.

  > 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
  > 19:47 < tobias-urdin> after a quick google "The problem was that my
CA DN was the same as the certificate DN."

IIRC I think that solved it, but then again I wouldn't remember fully
since I've been at so many different angles by now.

Here is my IRC logs history from the #openstack-lbaas channel, perhaps
it can help you out
http://paste.openstack.org/show/732575/


Tobias, I owe you a beer. This was precisely the issue. I'm deploying
Octavia with kolla-ansible. It only deploys a single CA. After hacking
the templates and playbook to incorporate a separate server CA, the
amphorae now load and provision the required namespace. I'm adding a
kolla tag to the subject of this in hopes that someone might want to
take on changing this behavior in the project. Hopefully after I get
through Upstream Institute in Berlin I'll be able to do it myself if
nobody else wants to do it.

For certificate generation, I extracted the contents of
octavia_certs_install.yml (which sets up the directory structure,
openssl.cnf, and the client CA), and octavia_certs.yml (which creates
the server CA and the client certificate) and mashed them into a
separate playbook just for this purpose. At the end I get:

ca_01.pem - Client CA Certificate
ca_01.key - Client CA Key
ca_server_01.pem - Server CA Certificate
cakey.pem - Server CA Key
client.pem - Concatenated Client Key and Certificate

If it would help to have the playbook, I can stick it up on github
with a huge "This is a hack" disclaimer on it.


-

Sorry for hijacking the thread but I'm stuck as well.

I've in the past tried to generate the certificates with [1] but now
moved on to using the openstack-ansible way of generating them [2]
with some modifications.

Right now I'm just getting: Could not connect to instance. Retrying.:
SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
from the amphoras, haven't got any further but I've eliminated a lot of
stuck in the middle.

Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
wasn't an issue with CentOS and OpenSSL versions since it tends to lag
behind.
Checking the amphora with openssl s_client [3] it gives the same one,
but the verification is successful just that I don't understand what the
bad signature
part is about, from browsing some OpenSSL code it seems to be related to
RSA signatures somehow.

140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
signature:s3_clnt.c:2032:

So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS
(openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm
back to something related
to the certificates or the communication between the endpoints, or what
actually responds inside the amphora (gunicorn IIUC?). Based on the
"verify" functions actually causing that bad signature error I would
assume it's the generated certificate that the amphora presents that is
causing it.

I'll have to continue the troubleshooting to the inside of the amphora,
I've used the test-only amphora image before but have now built my own
one that is
using the amphora-agent from the actual stable branch, but same issue
(bad signature).

For verbosity this is the config options set for the certificates in
octavia.conf and which file it was copied from [4], same here, a
replication of what openstack-ansible does.

Appreciate any feedback or help :)

Best regards
Tobias

[1]
https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh
[2] http://paste.openstack.org/show/732483/
[3] http://paste.openstack.org/show/732486/
[4] http://paste.openstack.org/show/732487/

On 10/20/2018 01:53 AM, Michael Johnson wrote:

Hi Erik,

Sorry to hear you are still having certificate issues.

Issue #2 is probably caused by issue #1. Since we hot-plug the tenant
network for the VIP, o

Re: [openstack-dev] [Octavia] [Kolla] SSL errors polling amphorae and missing tenant network interface

[Tobias Urdin]

Might as well throw it out here.

After a lot of troubleshooting we were able to narrow our issue down to 
our test environment running qemu virtualization, we moved our compute 
node to hardware and

used kvm full virtualization instead.

We could properly reproduce the issue where generating a CSR from a 
private key and then trying to verify the CSR would fail complaining about

"Signature did not match the certificate request"

We suspect qemu floating point emulation caused this, the same OpenSSL 
function that validates a CSR is the one used when validating the SSL 
handshake which caused our issue.
After going through the whole stack, we have Octavia working flawlessly 
without any issues at all.


Best regards
Tobias

On 10/23/2018 04:31 PM, Tobias Urdin wrote:

Hello Erik,

Could you specify the DNs you used for all certificates just so that I
can rule it out on my side.
You can redact anything sensitive with some to just get the feel on how
it's configured.

Best regards
Tobias

On 10/22/2018 04:47 PM, Erik McCormick wrote:

On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin  wrote:

Hello,

I've been having a lot of issues with SSL certificates myself, on my
second trip now trying to get it working.

Before I spent a lot of time walking through every line in the DevStack
plugin and fixing my config options, used the generate
script [1] and still it didn't work.

When I got the "invalid padding" issue it was because of the DN I used
for the CA and the certificate IIRC.

   > 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING
octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect
to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa
routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'),
('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'),
('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",)
   > 19:47 < tobias-urdin> after a quick google "The problem was that my
CA DN was the same as the certificate DN."

IIRC I think that solved it, but then again I wouldn't remember fully
since I've been at so many different angles by now.

Here is my IRC logs history from the #openstack-lbaas channel, perhaps
it can help you out
http://paste.openstack.org/show/732575/


Tobias, I owe you a beer. This was precisely the issue. I'm deploying
Octavia with kolla-ansible. It only deploys a single CA. After hacking
the templates and playbook to incorporate a separate server CA, the
amphorae now load and provision the required namespace. I'm adding a
kolla tag to the subject of this in hopes that someone might want to
take on changing this behavior in the project. Hopefully after I get
through Upstream Institute in Berlin I'll be able to do it myself if
nobody else wants to do it.

For certificate generation, I extracted the contents of
octavia_certs_install.yml (which sets up the directory structure,
openssl.cnf, and the client CA), and octavia_certs.yml (which creates
the server CA and the client certificate) and mashed them into a
separate playbook just for this purpose. At the end I get:

ca_01.pem - Client CA Certificate
ca_01.key - Client CA Key
ca_server_01.pem - Server CA Certificate
cakey.pem - Server CA Key
client.pem - Concatenated Client Key and Certificate

If it would help to have the playbook, I can stick it up on github
with a huge "This is a hack" disclaimer on it.


-

Sorry for hijacking the thread but I'm stuck as well.

I've in the past tried to generate the certificates with [1] but now
moved on to using the openstack-ansible way of generating them [2]
with some modifications.

Right now I'm just getting: Could not connect to instance. Retrying.:
SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579)
from the amphoras, haven't got any further but I've eliminated a lot of
stuck in the middle.

Tried deploying Ocatavia on Ubuntu with python3 to just make sure there
wasn't an issue with CentOS and OpenSSL versions since it tends to lag
behind.
Checking the amphora with openssl s_client [3] it gives the same one,
but the verification is successful just that I don't understand what the
bad signature
part is about, from browsing some OpenSSL code it seems to be related to
RSA signatures somehow.

140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad
signature:s3_clnt.c:2032:

So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS
(openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm
back to something related
to the certificates or the communication between the endpoints, or what
actually responds inside the amphora (gunicorn IIUC?). Based on the
"verify" functions actually causing that bad signature error I would
assume it's the generated certificate that the amphora presents that is
causing it.

I'll have to continue the troubleshooting to the inside of the amphora,
I've used the test-only amphora image before

[openstack-dev] [puppet] Heads up for changes causing restarts!

[Tobias Urdin]

Hello,

Due to bugs and fixes that has been needed we are probably going to 
merge some changes to
Puppet modules which will cause a refresh of their services meaning they 
will be restarted.


If you are following the stable branches (stable/rocky in this case) and 
not using tagged releases when you
are pulling in the Puppet OpenStack modules we want to alert you that 
restarts of services might happen

if you deploy new changes.

These two for example is bug fixes which are probably going to be 
restarted causing restart of Horizon

and Cinder services [1] [2] [3].

Feel free to reach out to us at #puppet-openstack if you have any concerns.

[1] https://review.openstack.org/#/c/608244/
[2] https://review.openstack.org/#/c/607964/ (if backported to Rocky 
later on)

[3] https://review.openstack.org/#/c/605071/

Best regards
Tobias

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [cinder][puppet][kolla][helm][ansible] Change in Cinder backup driver naming

[Tobias Urdin]

Thanks Sean!

I did a quick sanity check on the backup part in the puppet-cinder 
module and there is no opinionated

default value there which needs to be changed.

Best regards

On 09/27/2018 08:37 PM, Sean McGinnis wrote:

This probably applies to all deployment tools, so hopefully this reaches the
right folks.

In Havana, Cinder deprecated the use of specifying the module for configuring
backup drivers. Patch https://review.openstack.org/#/c/595372/ finally removed
the backwards compatibility handling for configs that still used the old way.

Looking through a quick search, it appears there may be some tools that are
still defaulting to setting the backup driver name using the patch. If your
project does not specify the full driver class path, please update these to do
so now.

Any questions, please reach out here or in the #openstack-cinder channel.

Thanks!
Sean


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] [stable] Deprecation of newton branches

[Tobias Urdin]

Hello,
This got lost way down in my mailbox.

I think we have a consensus about getting rid of the newton branches.
Does anybody in Stable release team have time to deprecate the 
stable/newton branches?


Best regards
Tobias

On 11/19/2018 06:21 PM, Alex Schultz wrote:

On Mon, Nov 19, 2018 at 1:18 AM Tobias Urdin  wrote:

Hello,

We've been talking for a while about the deprecation and removal of the
stable/newton branches.
I think it's time now that we get rid of them, we have no open patches
and Newton is considered EOL.

Could cores get back with a quick feedback and then the stable team can
get rid of those whenever they have time.


yes please. let's EOL them


Best regards
Tobias

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [puppet] [stable] Deprecation of newton branches

[Tobias Urdin]

Hello,

We've been talking for a while about the deprecation and removal of the 
stable/newton branches.
I think it's time now that we get rid of them, we have no open patches 
and Newton is considered EOL.


Could cores get back with a quick feedback and then the stable team can 
get rid of those whenever they have time.


Best regards
Tobias

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [puppet] [placement]

[Tobias Urdin]

Sounds great, thanks for pushing this Emilien!

Best regards
Tobias

On 09/15/2018 07:48 PM, Emilien Macchi wrote:

I'm currently taking care of creating puppet-placement:
https://review.openstack.org/#/c/602870/
https://review.openstack.org/#/c/602871/
https://review.openstack.org/#/c/602869/

Once these merge, we'll use cookiecutter, and move things from 
puppet-nova. We'll also find a way to call puppet-placement from 
nova::placement class, eventually.

Hopefully we can make the switch to new placement during Stein!

Thanks,
--
Emilien Macchi


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev