Re: [Openstack] missing ovs flows and extra interfaces in pike

[Fabian Zimmermann]

Hi,

you may contact me directly, this would speed up my responsetime ;)

Am 26.10.18 um 16:15 schrieb Hartwig Hauschild:

We thought we're missing the flow for "if you're looking for this MAC go
this way", but it turned out that what's actually missing is a bunch of
interfaces on the multicast-flow for the vlan that we're investigating.

Is that what you're seeing as well?


exactly.

I wrote a small script which uses the database to calculate which 
network is located on which HV and checks if there are suitable 
vxlan-tunnel build.


Its a quick and dirty hack, but works for us (so far).

I personally dont like publishing code in this quality, but I dont think 
I will improve this in the near future - so here maybe it helps you a bit.


https://github.com/noris-network/check_vxlan_mesh

If you have any questions, dont hesitate to ask.

 Fabian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] missing ovs flows and extra interfaces in pike

[Fabian Zimmermann]

Hi,

we see something similar. We are running ovs, vxlan - but legacy routers 
directly on hvs without any network nodes.


Currently we didnt find a way to fix or reproduce the issue. We just 
wrote a small script which calcuates which flow should be on with hv and 
if something is missing -> send an alert.


The operator will then restart the suitable ovs-agent and everything is 
working again.


We also found out, that the problem is gone as soon as we disable l2pop, 
but this is not possible if we (as you already did) switch to dvr.


So at the moment we plan to disable l2pop and move our routers back to 
some network nodes.


I would be glad if someone is able to reproduce or even better - fix the 
issue.


 Fabian

Am 19.10.18 um 15:32 schrieb Hartwig Hauschild:

Hi,

[ I have no idea how much of the following information is necessary ]

We're running Openstack Pike, deployed with Openstack-Ansible 16.0.5.
The system is running on a bunch of compute-nodes and three combined
network/management-nodes, we're using OVS, DVR and VXLAN for networking.

The DVRs are set up with snat disabled, that's handled by different
systems.

We have recently noticed that we don't have north-south-connectivity in
a couple of qdhcp-netns and after a weeks worth of debugging it boils
down to missing OVS-flows on br-tun that should be directing the
northbound traffic at the node with the live snat-netns.

We also noticed that while every node has the ports for the
qdhcp-netns that belong on the node we also have a couple of taps and
flows for ports that are on other nodes.

To make that a bit clearer:
If you have network A with dhcp-services F, G, H we found that the ip
netns containing the dnsmasq for F, G, H are on nodes 1, 2, 3
respectively, but node 1 would also have the tap-interface and flows for
G on br-int dangeling freely without any netns.

Is there a simple explanation for this and maybe even a fix?

What we found so far seems to suggest we should either restart the
management-nodes or the neutron-agent-containers or at least stop, clean
and start ovs and neutron-openvswitch-agent inside the containers.

Is it possible to somehow redeploy or validate the flows from neutron to
make sure that everything is consistent apart from restarts?




___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [libvirt][nova] What can be used as equivalent to libvirt network hooks file?

[Fabian Zimmermann]

Hi,

what are you trying to execute? Maybe there is a easier way to reach your goal?

f. e. a udev-trigger or something like listening for events in your 
message-queue.

 Fabian 

Am 12. Mai 2018 17:26:35 MESZ schrieb "Palin, Francois" 
<francois.pa...@windriver.com>:
>Hi Fabian,
>
>Yes, I have looked at hooks/qemu, and the xml data it receives doesn't
>have
>the information needed to be able to logically derive the network and
>interface.
>
>Thanks,
>
>François
>
>
>____
>From: Fabian Zimmermann [dev@gmail.com]
>Sent: Saturday, May 12, 2018 12:27 AM
>To: openstack@lists.openstack.org; Palin, Francois;
>openstack@lists.openstack.org
>Subject: Re: [Openstack] [libvirt][nova] What can be used as equivalent
>to libvirt network hooks file?
>
>Hi,
>
>did you try
>
>hooks/qemu?
>
>Fabian
>
>Am 11. Mai 2018 16:22:14 MESZ schrieb "Palin, Francois"
><francois.pa...@windriver.com>:
>Hi all,
>
>libvirt network hooks file ( /etc/libvirt/hooks/network ) does not get
>called when using neutron.
>Any suggestion as to what could be used instead?
>
>I need to perform some specific actions on an interface once an
>instance releases it.
>
>Thanks,
>
>François
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [libvirt][nova] What can be used as equivalent to libvirt network hooks file?

[Fabian Zimmermann]

Hi,

did you try

hooks/qemu?

 Fabian 

Am 11. Mai 2018 16:22:14 MESZ schrieb "Palin, Francois" 
:
>Hi all,
>
>libvirt network hooks file ( /etc/libvirt/hooks/network ) does not get
>called when using neutron.
>Any suggestion as to what could be used instead?
>
>I need to perform some specific actions on an interface once an
>instance releases it.
>
>Thanks,
>
>François
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Ocata update for Pike on Ubuntu

[Fabian Zimmermann]

Hi,

Am 25.04.2018 um 15:05 schrieb Marcio Prado:

The mistake is in Keystone. I have not had time to debug yet ...

[authz_core:error] client denied by server configuration: 
/usr/bin/kestone-wsgi-public


first thought, sounds like apache-config is blocking the access.

Check your apache(2)/sites-enabled/


 Fabian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Ocata update for Pike on Ubuntu

[Fabian Zimmermann]

Hi,

Am 25.04.2018 um 13:30 schrieb Marcio Prado:

I'm also simulating the update ... Unsuccessfully.


What issues are you running into?

 Fabian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Ocata update for Pike on Ubuntu

[Fabian Zimmermann]

Hi,

Am 24.04.2018 um 23:58 schrieb Marcio Prado:

Has anyone upgraded OpenStack Ocata to Pike using Ubuntu 16.04?


we are currently evaluating/simulating the upgrade


Can you share the experiences please?


well, until now we found

https://bugs.launchpad.net/ubuntu/+source/cinder/+bug/1766189

but thats not really an upgrade issue, it is a pike issue.

 Fabian

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] about cloud-init question

[Fabian Zimmermann]

Hi,

I dont think its possible to switch the interpreter during the run.

I would suggest to take a look into your cloud-init.log maybe its telling you 
why it is not doing what you want.

 Fabian Zimmermann


Am 23. April 2018 03:08:15 MESZ schrieb "Huang, Haibin" 
<haibin.hu...@intel.com>:
>Hi All,
>
>I have a problem about cloud-init.
>I want to both transfer files and execute script. So I give below
>script to user-data when I create instance.
>#cloud-config
>write_files:
>-   encoding: b64
>content: H4sICMxh2VoAA2hoYgCzKE5JK07hAgDCo1pOBw==
>owner: root:root
>path: /root/hhb.gz
>permissions: '0644'
>
>#!/bin/bash
>mkdir -p /home/ubuntu/config
>
>but, I can't get /root/hhb.gz and /home/Ubuntu/config.
>If I separate transfer files and execute script. It is ok.
>Any idea?
>
>Below is my debug info
>
>ubuntu@onap-hhb7:~$ sudo cloud-init --version
>
>sudo: unable to resolve host onap-hhb7
>
>cloud-init 0.7.5
>
>
>
>security-groupsubuntu@onap-hhb7:~$ curl 
>http://169.254.169.254/2009-04-04/user-data
>
>#cloud-config
>
>write_files:
>
>-   encoding: b64
>
>content: H4sICMxh2VoAA2hoYgCzKE5JK07hAgDCo1pOBw==
>
>owner: root:root
>
>path: /root/hhb.gz
>
>permissions: '0644'
>
>
>
>#!/bin/bash
>
>mkdir -p /home/ubuntu/config
>
>
>
>ubuntu@onap-hhb7:~$ sudo ls /root/ -a
>
>.  ..  .bashrc  .profile  .ssh
>
>
>
>ubuntu@onap-hhb7:/var/lib/cloud/instance$ ls
>
>boot-finished datasource  obj.pkl  semuser-data.txt.i 
>vendor-data.txt.i
>
>cloud-config.txt  handlersscripts  user-data.txt  vendor-data.txt
>
>ubuntu@onap-hhb7:/var/lib/cloud/instance$ sudo cat user-data.txt
>
>sudo: unable to resolve host onap-hhb7
>
>#cloud-config
>
>write_files:
>
>-   encoding: b64
>
>content: H4sICMxh2VoAA2hoYgCzKE5JK07hAgDCo1pOBw==
>
>owner: root:root
>
>path: /root/hhb.gz
>
>permissions: '0644'
>
>
>
>#!/bin/bash
>
>mkdir -p /home/ubuntu/config
>
>
>
>---
>Huang.haibin
>11628530
>86+18106533356
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Glance image definition using V2 API

[Fabian Zimmermann]

Hi,

just create an empty image (Without file or location param), then use 
add-location to set your locations.

 Fabian Zimmermann


Am 19. April 2018 06:15:33 MESZ schrieb Cory Hawkless <c...@hawkless.id.au>:
>Looking for some help with defining glance images. I'm running a new
>Queens installation and do not have the V1 API enabled in Glance.
>
>So the Glance V1 API has been deprecated for some time now (I believe)
>and best I can tell there is no support in the V2 API for defining an
>existing image into glance.
>I.E, I have some volumes in my Ceph pool that I'd like to expose to
>Glance, but the old method of using "glance image-create --disk-format
>raw --id $IMAGE_ID  --location rbd://$CLUSTER_ID/$POOL/$IMAGE_ID/snap"
>no longer works because this is a V1 command with the V2 API having no
>support for the --location flag.
>
>I'm primarily dealing with large(ish) windows images around 100GB mark,
>so exporting them to a file then importing them using the --file
>command is very sub optimal.
>
>Without an outright database hack, is there any way to define an
>existing Ceph based volume to be used by Glance?
>If there is not a way to do this then can I safely enable the V1 API in
>Queens? How long until V1 support is removed and I'm back to square 1
>
>Thanks in advance
>Cory
>
>___
>Mailing list:
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>Post to : openstack@lists.openstack.org
>Unsubscribe :
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] Intro and Containerized Control Plane

[Fabian Zimmermann]

Hi,

we run completely in containers. I would recommend to take a look at how kolla 
is creating and managing the containers.

This should prevent you from the bigger pitfalls :)

If you have any specific questions. Don't hesitate to ask.

Fabian Zimmermann

Am 20. April 2018 17:22:46 MESZ schrieb Michael Damkot <mdam...@salesforce.com>:
>Hello Operators!!
>
>I wanted to say "Hello" to the community once again! I've come back
>into
>the OpenStack fold after my time as a former member of the Time Warner
>Cable Team.
>
>Salesforce is working toward greatly increasing the size and scale of
>our
>OpenStack use cases as well as our participation in the community.
>We're
>currently deep diving on a few things including containerizing a number
>of
>control plane components. Is anyone willing to share any hurdles or
>hiccups
>they've hit while exploring containerization? I didn't see much of
>anything
>in the archives but I know we aren't the only ones heading down this
>path.
>
>Thanks in advance!
>
>--
>Michael Damkot
>@mdamkot - twitter
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack] [openstack][pike][cinder] cinder-volume agent is marked down during image upload

[Fabian Zimmermann]

Hi,

we just evaluating the upgrade to pike and running into an issue with 
cinder-volume agent


We see cinder-volume-Agent being marked DOWN if we request an 
volume-upload to glance and the upload runs longer than heartbeat-timeout.


As soon as curl starts to upload the image, there are no further 
heartbeat-updates in the database.


steps to reproduce:

1. create a big Volume (f.e. 100G)
2. fill the volume with data to avoid too fast upload
3. tell cinder to upload the volume as image
4. watch cinder-volume-agent being marked as DOWN, because no 
heartbeat-update is done during upload.


Is anybody experiencing the same issue?

Thanks a lot,

 Fabian


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack-operators] sporadic missing vxlan-tunnel-port assignment

[Fabian Zimmermann]

Hi,

we currently see sporadic communication problems.

After some research we found out, that this is caused by missing 
tunnel-port assignments in table 21 of openvswitch.


Today we had the issue again and here the logs of the add_fdb_entries 
calls at the affected system:


neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent 
method add_fdb_entries called with arguments (object at 0x7f7264509810>,) {u'fdb_entries': 
{u'cd2baf3d-427c-41be-be56-7cbb8176067f': {u'segment_id': 96, u'ports': 
{u'10.78.23.12': [[u'00:00:00:00:00:00', u'0.0.0.0'], 
[u'fa:16:3e:d0:a0:77', u'192.168.0.2']]}, u'network_type': u'vxlan'}}}


neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent 
method add_fdb_entries called with arguments (object at 0x7f7264509bd0>,) {u'fdb_entries': 
{u'cd2baf3d-427c-41be-be56-7cbb8176067f': {u'segment_id': 96, u'ports': 
{u'10.78.23.11': [[u'00:00:00:00:00:00', u'0.0.0.0'], 
[u'fa:16:3e:29:0c:d5', u'192.168.0.3']]}, u'network_type': u'vxlan'}}}


neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent 
method add_fdb_entries called with arguments (object at 0x7f7264509d90>,) {u'fdb_entries': 
{u'cd2baf3d-427c-41be-be56-7cbb8176067f': {u'segment_id': 96, u'ports': 
{u'10.78.12.101': []}, u'network_type': u'vxlan'}}}


neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent 
method add_fdb_entries called with arguments (object at 0x7f7264627190>,) {u'fdb_entries': 
{u'cd2baf3d-427c-41be-be56-7cbb8176067f': {u'segment_id': 96, u'ports': 
{u'10.79.20.102': [[u'00:00:00:00:00:00', u'0.0.0.0']]}, 
u'network_type': u'vxlan'}}}


The missing tunnel-port is the connection to 10.78.12.101, it looks like 
the empty array/dict may cause this issue.


Any hints how to further debug the situation?

What may cause an empty dict in add_fdb_entries?

Thanks a lot,


 Fabian Zimmermann

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators