Re: [Openstack] [Openstack-operators] Recovering from full outage

[John Petrini]

You might want to try giving the neutron-dhcp and metadata agents a restart.
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] [Openstack] Recovering from full outage

[John Petrini]

You might want to try giving the neutron-dhcp and metadata agents a restart.
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack] [Openstack-operators] Recovering from full outage

[John Petrini]

Are you instances receiving a route to the metadata service
(169.254.169.254) from DHCP? Can you curl the endpoint? curl
http://169.254.169.254/latest/meta-data
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] [Openstack] Recovering from full outage

[John Petrini]

Are you instances receiving a route to the metadata service
(169.254.169.254) from DHCP? Can you curl the endpoint? curl
http://169.254.169.254/latest/meta-data
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack] HA Compute & Instance Evacuation

[John Petrini]

Take this with a grain of salt because we're using the original version
before the project moved under the Big Tent and I'm not sure how much it's
evolved since then. I assume the basic functions are the same though.

You're correct; Corosync and Pacemaker are used to determine if a compute
node goes down. The masakari-host-monitor process runs on each compute node
and checks the cluster status and sends a notification to
masakari-controller when a node goes down. The controller process keeps a
list of reserved hosts in it's database and calls nova host-evacuate to
move the Instances to one of the reserved hosts.

In our environment I also configured STONITH and I'd highly recommend it.
With STONITH Pacemaker sends a shutdown command to the Out of Band
Management card of the unreachable node to make sure that it can't come
back and cause a conflict.

There are two other components, masakari-process-monitor and
masakari-instance-monitor. These also run on your compute nodes. The former
watches the nova-compute service and the later monitors running instances
and restarts them if necessary.

Looking here it seems they've split Masakari into thee different repos:
https://github.com/openstack?utf8=%E2%9C%93=masakari==

masakari - The controller service and API
masakari-monitors - Compute node monitoring services
python-masakari-client - The cli tools
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] HA Compute & Instance Evacuation

[John Petrini]

We're using the original Masakari project for this and it works really
well. In fact just last week we lost a compute node and all of VM's were
successfully migrated to a reserve host in under 5 minutes. It's a really
nice feeling when your infrastructure heals itself before you even get a
chance to start troubleshooting.

It does require a good deal of configuration to get it up and running,
especially the clustering with Pacemaker/Corosync so be prepared to get
familiar with those tools and STONITH if you're not already. Worth it if
some of your infrastructure doesn't have redundancy built in at higher
level.
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Fuel 9.2 - Need to Scale Out Controllers

[John Petrini]

I've never done it myself and I strongly suggest you test before doing this
on a production environment but the general process would be to bootstrap
two new nodes, assign them the controller role in Fuel and click deploy
changes. This will kick of a puppet run on all nodes that will make the
necessary changes.

On Thu, Nov 30, 2017 at 3:17 AM, Raja T Nair  wrote:

> Hello All,
>
> Can someone please guide me on this?
>
> Regards,
> Raja.
>
> On 28 November 2017 at 13:48, Raja T Nair  wrote:
>
>> Hello All,
>>
>> Fuel - 9.2
>> Controller and Compute on Ubuntu 14.04
>>
>> I'm currently running a cloud with only one controller.
>> This was done because we had an urgent requirement for compute and
>> suitable hardware was
>>  not available at the time.
>> Now we want to add 2 more controllers to have proper redundancy in place.
>>
>> Please suggest me on what is the best way to achieve this.
>>
>> Best Regards,
>> Raja.
>>
>>
>> --
>> :^)
>>
>
>
>
> --
> :^)
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Mirantis][Fuel-9.0] UI not accessible

[John Petrini]

Is nginx running?

On Tue, Oct 17, 2017 at 8:31 AM, Raja T Nair  wrote:

> Hello All,
>
> I downloaded the latest Mirantis Openstack iso from the website and
> installed it as a virtualbox vm.
>
> Installation went through without any issue. the VM can communicate to the
> outside world using given IP: 10.10.31.2/24
>
> But the web UI is not accessible. When I try to access it through a
> browser, it redirects me to port 8443 , which errors out.
>
> I checked in the fuel master node, and port 8443 is not listening there.
> Is there any more steps to perform after the installation, that brings the
> webUI up?
>
> Can someone please help me troubleshoot this? I am doing fuel based
> installation for the first time.
>
> Regards,
> Raja.
> --
> :^)
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] extend attached volumes

[John Petrini]

I think this feature is actually implemented in nova. So you have to use
the nova volume-extend option to do what you want. This is just my
interpretation of the release notes though. I haven't tried it.


On Tue, Sep 26, 2017 at 10:20 AM, Volodymyr Litovka  wrote:

> Hi Jay,
>
> I know about this way :-) but Pike introduced ability to resize attached
> volumes:
>
> "It is now possible to signal and perform an online volume size change as
> of the 2.51 microversion using the volume-extendedexternal event. Nova
> will perform the volume extension so the host can detect its new size. It
> will also resize the device in QEMU so instance can detect the new disk
> size without rebooting." -- https://docs.openstack.org/
> releasenotes/nova/pike.html
>
> On 9/26/17 5:04 PM, Jay Pipes wrote:
>
> Detach the volume, then resize it, then re-attach.
>
> Best,
> -jay
>
> On 09/26/2017 09:22 AM, Volodymyr Litovka wrote:
>
> Colleagues,
>
> can't find ways to resize attached volume. I'm on Pike.
>
> As far as I understand, it required to be supported in Nova, because
> Cinder need to check with Nova whether it's possible to extend this volume.
>
> Well,
> - Nova's API microversion is 2.51, which seems to be enough to support
> "volume-extended" API call
> - Properties of image are *hw_disk_bus='scsi'* and
> *hw_scsi_model='virtio-scsi'*, type bare/raw, located in Cinder
> - hypervisor is KVM
> - volume is bootable, mounted as root, created as snapshot from Cinder
> volume
> - Cinder's backend is CEPH/Bluestore
>
> and both "cinder extend" and "openstack volume set --size" returns "Volume
> status must be '{'status': 'available'}' to extend, currently in-use".
>
> I did not find any configuration options neither in nova nor in cinder
> config files, which can help with this functionality.
>
> What I'm doing wrong?
>
> Thank you.
>
> --
> Volodymyr Litovka
>"Vision without Execution is Hallucination." -- Thomas Edison
>
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
> --
> Volodymyr Litovka
>   "Vision without Execution is Hallucination." -- Thomas Edison
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] MTU on Provider Networks

[John Petrini]

Great! Thank you both for the information.

John Petrini
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] [Openstack] MTU on Provider Networks

[John Petrini]

Great! Thank you both for the information.

John Petrini
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] MTU on Provider Networks

[John Petrini]

Hi List,

We are running Mitaka and having an MTU issue. Instances that we launch on
our provider network use Jumbo Frames (9000 MTU). There is a Layer2 link
between the OpenStack switches and our Core. This link uses and MTU of
1500.

Up until recently this MTU mismatch has not been an issue because none of
our systems are sending large enough packets to cause a problem. Recently
we've begun implementing a SIP device that sends very large packets,
sometimes even over 9000 bytes and requires fragmentation.

What we found in our troubleshooting is that when large packets originate
from our network to an instance in OpenStack they are being fragmented (as
expected). Once these packets reach the qbr-XX port iptables
defragments the packet and forwards it to the tap interface unfragmented.
If we set the MTU on the tap interface to 1500 it will refragment the
packet before forwarding it to the instance.

A similar issue happens the other direction. Large packets originating from
the OpenStack instance are fragmented (we set the mtu of the interface in
the instance to 1500 so this is expected) but again once the packets reach
the qbr--XX interface iptables defragments them again. If we set
the MTU of the qvb-XX to 1500 the packet is refragmented.

So long story short if we set the instance MTU to 1500 and the
qbr-XX and qvb-XX ports on the compute node to 1500 MTU the
packets remain fragmented and are able to traverse the network.

So the question becomes can we modify the default MTU of our provider
networks so that the instances created on this network receive a 1500 MTU
from DHCP and the ports on the compute node are also configured to a 1500
MTU?

I've been looking at the following neutron config option in
/etc/neutron/plugins/ml2/ml2_conf.ini:

physical_network_mtus =physnet1:9000,providernet:9000

Documentation on this setting is not very clear. Will adjusting this to
1500 for providernet accomplish what we need?

Thank You,

John Petrini
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack] MTU on Provider Networks

[John Petrini]

Hi List,

We are running Mitaka and having an MTU issue. Instances that we launch on
our provider network use Jumbo Frames (9000 MTU). There is a Layer2 link
between the OpenStack switches and our Core. This link uses and MTU of
1500.

Up until recently this MTU mismatch has not been an issue because none of
our systems are sending large enough packets to cause a problem. Recently
we've begun implementing a SIP device that sends very large packets,
sometimes even over 9000 bytes and requires fragmentation.

What we found in our troubleshooting is that when large packets originate
from our network to an instance in OpenStack they are being fragmented (as
expected). Once these packets reach the qbr-XX port iptables
defragments the packet and forwards it to the tap interface unfragmented.
If we set the MTU on the tap interface to 1500 it will refragment the
packet before forwarding it to the instance.

A similar issue happens the other direction. Large packets originating from
the OpenStack instance are fragmented (we set the mtu of the interface in
the instance to 1500 so this is expected) but again once the packets reach
the qbr--XX interface iptables defragments them again. If we set
the MTU of the qvb-XX to 1500 the packet is refragmented.

So long story short if we set the instance MTU to 1500 and the
qbr-XX and qvb-XX ports on the compute node to 1500 MTU the
packets remain fragmented and are able to traverse the network.

So the question becomes can we modify the default MTU of our provider
networks so that the instances created on this network receive a 1500 MTU
from DHCP and the ports on the compute node are also configured to a 1500
MTU?

I've been looking at the following neutron config option in
/etc/neutron/plugins/ml2/ml2_conf.ini:

physical_network_mtus =physnet1:9000,providernet:9000

Documentation on this setting is not very clear. Will adjusting this to
1500 for providernet accomplish what we need?

Thank You,

John Petrini
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] Pacemaker / Corosync in guests on OpenStack

[John Petrini]

I just did recently and had no issues. I used a provider network so I don't
have experience using it with project networks but I believe the only issue
you might run into with project networks is multicast. You can work around
this by using unicast instead.

If you do you use multicast you need to enable IGMP in your security
groups. You can do this in Horizon by selecting other protocol and setting
the IP protocol number to 2.

I hit a minor issue setting up a VIP because port security wouldn't allow
traffic to the instance that was destined for that address but all I had to
do was add the VIP as an allowed address pair on the port of each instance.
Also, I attached an additional interface to one of the instances to
allocate the VIP, I just didn't configure the interface within the
instance. Since we use DHCP this was a simple way to reserve the IP. I'm
sure I could have created a pacemaker resource that would move the port
using the OpenStack API but I prefer the simplicity and speed of Pacemakers
ocf:ipaddr2 resource.

I setup fencing of the instances via the openstack api to avoid any chance
of a duplicate IP when moving the VIP. I borrowed this script
https://github.com/beekhof/fence_openstack/blob/master/fence_openstack and
made a few minor changes.

Overall there weren't many differences between setting up pacemaker in
OpenStack vs Iron but I hope this is helpful.


Regards,

John Petrini


On Wed, Aug 16, 2017 at 6:06 AM, Tim Bell <tim.b...@cern.ch> wrote:

>
>
> Has anyone had experience setting up a cluster of VM guests running
> Pacemaker / Corosync? Any recommendations?
>
>
>
> Tim
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Experience with Cinder volumes as root disks?

[John Petrini]

Thanks for the info. Might have something to do with the Ceph version then.
We're running hammer and apparently the du option wasn't added until
in Infernalis.

John Petrini

On Tue, Aug 1, 2017 at 4:32 PM, Mike Lowe <joml...@iu.edu> wrote:

> Two things, first info does not show how much disk is used du does.
> Second, the semantics count, copy is different than clone and flatten.
> Clone and flatten which should happen if you have things working correctly
> is much faster than copy.  If you are using copy then you may be limited by
> the number of management ops in flight, this is a setting for more recent
> versions of ceph.  I don’t know if copy skips zero byte objects but clone
> and flatten certainly do.  You need to be sure that you have the proper
> settings in nova.conf for discard/unmap as well as using
> hw_scsi_model=virtio-scsi and hw_disk_bus=scsi in the image properties.
> Once discard is working and you have the qemu guest agent running in your
> instances you can force them to do a fstrim to reclaim space as an
> additional benefit.
>
> On Aug 1, 2017, at 3:50 PM, John Petrini <jpetr...@coredial.com> wrote:
>
> Maybe I'm just not understanding but when I create a nova snapshot the
> snapshot happens at RBD in the ephemeral pool and then it's copied to the
> images pool. This results in a full sized image rather than a snapshot with
> a reference to the parent.
>
> For example below is a snapshot of an ephemeral instance from our images
> pool. It's 80GB, the size of the instance, so rather than just capturing
> the state of the parent image I end up with a brand new image of the same
> size. It takes a long time to create this copy and causes high IO during
> the snapshot.
>
> rbd --pool images info d5404709-cb86-4743-b3d5-1dc7fba836c1
> rbd image 'd5404709-cb86-4743-b3d5-1dc7fba836c1':
> size 81920 MB in 20480 objects
> order 22 (4096 kB objects)
> block_name_prefix: rbd_data.93cdd43ca5efa8
> format: 2
> features: layering, striping
> flags:
> stripe unit: 4096 kB
> stripe count: 1
>
>
> John Petrini
>
> On Tue, Aug 1, 2017 at 3:24 PM, Mike Lowe <joml...@iu.edu> wrote:
>
>> There is no upload if you use Ceph to back your glance (like you should),
>> the snapshot is cloned from the ephemeral pool into the the images pool,
>> then flatten is run as a background task.  Net result is that creating a
>> 120GB image vs 8GB is slightly faster on my cloud but not at all what I’d
>> call painful.
>>
>> Running nova image-create for a 8GB image:
>>
>> real 0m2.712s
>> user 0m0.761s
>> sys 0m0.225s
>>
>> Running nova image-create for a 128GB image:
>>
>> real 0m2.436s
>> user 0m0.774s
>> sys 0m0.225s
>>
>>
>>
>>
>> On Aug 1, 2017, at 3:07 PM, John Petrini <jpetr...@coredial.com> wrote:
>>
>> Yes from Mitaka onward the snapshot happens at the RBD level which is
>> fast. It's the flattening and uploading of the image to glance that's the
>> major pain point. Still it's worlds better than the qemu snapshots to the
>> local disk prior to Mitaka.
>>
>> John Petrini
>>
>> Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
>> Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
>> <http://www.linkedin.com/company/99631>   [image: Google Plus]
>> <https://plus.google.com/104062177220750809525/posts>   [image: Blog]
>> <http://success.coredial.com/blog>
>> 751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
>> *P:* 215.297.4400 x232 <(215)%20297-4400>   //   *F: *215.297.4401
>> <(215)%20297-4401>   //   *E: *jpetr...@coredial.com
>> <https://t.xink.io/Tracking/Index/a64BAGORAACukSoA0>
>>
>> The information transmitted is intended only for the person or entity to
>> which it is addressed and may contain confidential and/or privileged
>> material. Any review, retransmission,  dissemination or other use of, or
>> taking of any action in reliance upon, this information by persons or
>> entities other than the intended recipient is prohibited. If you received
>> this in error, please contact the sender and delete the material from any
>> computer.
>>
>>
>>
>> On Tue, Aug 1, 2017 at 2:53 PM, Mike Lowe <joml...@iu.edu> wrote:
>>
>>> Strictly speaking I don’t think this is the case anymore for Mitaka or
>>> later.  Snapping nova does take more space as the image is flattened, but
>>> the dumb download then upload back into ceph has been cut out.  With
>>> careful attention paid to discard/TRIM I believe you can maintain the thin
>>

Re: [Openstack-operators] Experience with Cinder volumes as root disks?

[John Petrini]

Maybe I'm just not understanding but when I create a nova snapshot the
snapshot happens at RBD in the ephemeral pool and then it's copied to the
images pool. This results in a full sized image rather than a snapshot with
a reference to the parent.

For example below is a snapshot of an ephemeral instance from our images
pool. It's 80GB, the size of the instance, so rather than just capturing
the state of the parent image I end up with a brand new image of the same
size. It takes a long time to create this copy and causes high IO during
the snapshot.

rbd --pool images info d5404709-cb86-4743-b3d5-1dc7fba836c1
rbd image 'd5404709-cb86-4743-b3d5-1dc7fba836c1':
size 81920 MB in 20480 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.93cdd43ca5efa8
format: 2
features: layering, striping
flags:
stripe unit: 4096 kB
stripe count: 1


John Petrini

On Tue, Aug 1, 2017 at 3:24 PM, Mike Lowe <joml...@iu.edu> wrote:

> There is no upload if you use Ceph to back your glance (like you should),
> the snapshot is cloned from the ephemeral pool into the the images pool,
> then flatten is run as a background task.  Net result is that creating a
> 120GB image vs 8GB is slightly faster on my cloud but not at all what I’d
> call painful.
>
> Running nova image-create for a 8GB image:
>
> real 0m2.712s
> user 0m0.761s
> sys 0m0.225s
>
> Running nova image-create for a 128GB image:
>
> real 0m2.436s
> user 0m0.774s
> sys 0m0.225s
>
>
>
>
> On Aug 1, 2017, at 3:07 PM, John Petrini <jpetr...@coredial.com> wrote:
>
> Yes from Mitaka onward the snapshot happens at the RBD level which is
> fast. It's the flattening and uploading of the image to glance that's the
> major pain point. Still it's worlds better than the qemu snapshots to the
> local disk prior to Mitaka.
>
> John Petrini
>
> Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
> Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
> <http://www.linkedin.com/company/99631>   [image: Google Plus]
> <https://plus.google.com/104062177220750809525/posts>   [image: Blog]
> <http://success.coredial.com/blog>
> 751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
> *P:* 215.297.4400 x232 <(215)%20297-4400>   //   *F: *215.297.4401
> <(215)%20297-4401>   //   *E: *jpetr...@coredial.com
> <https://t.xink.io/Tracking/Index/a64BAGORAACukSoA0>
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission,  dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
>
>
> On Tue, Aug 1, 2017 at 2:53 PM, Mike Lowe <joml...@iu.edu> wrote:
>
>> Strictly speaking I don’t think this is the case anymore for Mitaka or
>> later.  Snapping nova does take more space as the image is flattened, but
>> the dumb download then upload back into ceph has been cut out.  With
>> careful attention paid to discard/TRIM I believe you can maintain the thin
>> provisioning properties of RBD.  The workflow is explained here.
>> https://www.sebastien-han.fr/blog/2015/10/05/openstack-nova
>> -snapshots-on-ceph-rbd/
>>
>> On Aug 1, 2017, at 11:14 AM, John Petrini <jpetr...@coredial.com> wrote:
>>
>> Just my two cents here but we started out using mostly Ephemeral storage
>> in our builds and looking back I wish we hadn't. Note we're using Ceph as a
>> backend so my response is tailored towards Ceph's behavior.
>>
>> The major pain point is snapshots. When you snapshot an nova volume an
>> RBD snapshot occurs and is very quick and uses very little additional
>> storage, however the snapshot is then copied into the images pool and in
>> the process is converted from a snapshot to a full size image. This takes a
>> long time because you have to copy a lot of data and it takes up a lot of
>> space. It also causes a great deal of IO on the storage and means you end
>> up with a bunch of "snapshot images" creating clutter. On the other hand
>> volume snapshots are near instantaneous without the other drawbacks I've
>> mentioned.
>>
>> On the plus side for ephemeral storage; resizing the root disk of images
>> works better. As long as your image is configured properly it's just a
>> matter of initiating a resize and letting the instance reboot to grow the
>> root disk. When using volumes as your root disk you instead have to
>> s

Re: [Openstack-operators] Experience with Cinder volumes as root disks?

[John Petrini]

Yes from Mitaka onward the snapshot happens at the RBD level which is fast.
It's the flattening and uploading of the image to glance that's the major
pain point. Still it's worlds better than the qemu snapshots to the local
disk prior to Mitaka.

John Petrini

Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
*P:* 215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com
<https://t.xink.io/Tracking/Index/a64BAGORAACukSoA0>

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.



On Tue, Aug 1, 2017 at 2:53 PM, Mike Lowe <joml...@iu.edu> wrote:

> Strictly speaking I don’t think this is the case anymore for Mitaka or
> later.  Snapping nova does take more space as the image is flattened, but
> the dumb download then upload back into ceph has been cut out.  With
> careful attention paid to discard/TRIM I believe you can maintain the thin
> provisioning properties of RBD.  The workflow is explained here.
> https://www.sebastien-han.fr/blog/2015/10/05/openstack-
> nova-snapshots-on-ceph-rbd/
>
> On Aug 1, 2017, at 11:14 AM, John Petrini <jpetr...@coredial.com> wrote:
>
> Just my two cents here but we started out using mostly Ephemeral storage
> in our builds and looking back I wish we hadn't. Note we're using Ceph as a
> backend so my response is tailored towards Ceph's behavior.
>
> The major pain point is snapshots. When you snapshot an nova volume an RBD
> snapshot occurs and is very quick and uses very little additional storage,
> however the snapshot is then copied into the images pool and in the process
> is converted from a snapshot to a full size image. This takes a long time
> because you have to copy a lot of data and it takes up a lot of space. It
> also causes a great deal of IO on the storage and means you end up with a
> bunch of "snapshot images" creating clutter. On the other hand volume
> snapshots are near instantaneous without the other drawbacks I've mentioned.
>
> On the plus side for ephemeral storage; resizing the root disk of images
> works better. As long as your image is configured properly it's just a
> matter of initiating a resize and letting the instance reboot to grow the
> root disk. When using volumes as your root disk you instead have to
> shutdown the instance, grow the volume and boot.
>
> I hope this help! If anyone on the list knows something I don't know
> regarding these issues please chime in. I'd love to know if there's a
> better way.
>
> Regards,
>
> John Petrini
>
> On Tue, Aug 1, 2017 at 10:50 AM, Kimball, Conrad <
> conrad.kimb...@boeing.com> wrote:
>
>> In our process of standing up an OpenStack internal cloud we are facing
>> the question of ephemeral storage vs. Cinder volumes for instance root
>> disks.
>>
>>
>>
>> As I look at public clouds such as AWS and Azure, the norm is to use
>> persistent volumes for the root disk.  AWS started out with images booting
>> onto ephemeral disk, but soon after they released Elastic Block Storage and
>> ever since the clear trend has been to EBS-backed instances, and now when I
>> look at their quick-start list of 33 AMIs, all of them are EBS-backed.  And
>> I’m not even sure one can have anything except persistent root disks in
>> Azure VMs.
>>
>>
>>
>> Based on this and a number of other factors I think we want our user
>> normal / default behavior to boot onto Cinder-backed volumes instead of
>> onto ephemeral storage.  But then I look at OpenStack and its design point
>> appears to be booting images onto ephemeral storage, and while it is
>> possible to boot an image onto a new volume this is clumsy (haven’t found a
>> way to make this the default behavior) and we are experiencing performance
>> problems (that admittedly we have not yet run to ground).
>>
>>
>>
>> So …
>>
>> · Are other operators routinely booting onto Cinder volumes
>> instead of ephemeral storage?
>>
>> · What has been your experience with this; any adv

Re: [Openstack] [Openstack-operators] UDP Buffer Filling

[John Petrini]

Hi Liping,

Thank you for the detailed response! I've gone over our environment and
checked the various values.

First I found that we are dropping packets on the physcial nics as well as
inside the instance (though only when its UDP receive buffer overflows).

Our physical nics are using the default ring size and our tap interfaces
are using the 500 tx_txdefault. There are dropped packets on the tap
interfaces but the counts are rather low and don't seem to increase very
often so I'm not sure that there's a problem there but I'm considering
adjusting the value anyway to avoid issues in the future.

We already tune these values in the VM. Would you suggest tuning them on
the compute nodes as well?
net.core.rmem_max / net.core.rmem_default / net.core.wmem_max /
net.core.rmem_default

I'm going to do some testing with multiqueues enabled since both you and
Saverio have suggested it.



John Petrini

Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
*P:* 215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

On Fri, Jul 28, 2017 at 9:25 AM, Erik McCormick <emccorm...@cirrusseven.com>
wrote:

>
>
> On Jul 28, 2017 8:51 AM, "John Petrini" <jpetr...@coredial.com> wrote:
>
> Hi Saverio,
>
> Thanks for the info. The parameter is missing completely:
>
> 
>   
>   
>   
>   
>function='0x0'/>
> 
>
> I've came across the blueprint for adding the image property
> hw_vif_multiqueue_enabled. Do you know if this feature is available in
> Mitaka?
>
> It was merged 2 years ago so should have been there since Liberty.
>
>
> John Petrini
>
> Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
> Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
> <http://www.linkedin.com/company/99631>   [image: Google Plus]
> <https://plus.google.com/104062177220750809525/posts>   [image: Blog]
> <http://success.coredial.com/blog>
>
> 751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
> *P:* 215.297.4400 x232 <(215)%20297-4400>   //   *F: *215.297.4401
> <(215)%20297-4401>   //   *E: *jpetr...@coredial.com
>
>
> On Fri, Jul 28, 2017 at 3:59 AM, Saverio Proto <ziopr...@gmail.com> wrote:
>
>> Hello John,
>>
>> a common problem is packets being dropped when they pass from the
>> hypervisor to the instance. There is bottleneck there.
>>
>> check the 'virsh dumpxml' of one of the instances that is dropping
>> packets. Check for the interface section, should look like:
>>
>> 
>>   
>>   
>>   
>>   
>>   
>>   
>>   > function='0x0'/>
>> 
>>
>> how many queues you have ??? Usually if you have only 1 or if the
>> parameter is missing completely is not good.
>>
>> in Mitaka nova should use 1 queue for every instance CPU core you
>> have. It is worth to check if this is set correctly in your setup.
>>
>> Cheers,
>>
>> Saverio
>>
>>
>>
>> 2017-07-27 17:49 GMT+02:00 John Petrini <jpetr...@coredial.com>:
>> > Hi List,
>> >
>> > We are running Mitaka with VLAN provider networking. We've recently
>> > encountered a problem where the UDP receive queue on instances is
>> filling up
>> > and we begin dropping packets. Moving instances out of OpenStack onto
>> bare
>> > metal resolves the issue completely.
>> >
>> > These instances are running asterisk which should be pulling these
>> packets
>> > off the queue but it appears to be falling behind no matter the
>> resources we
>> > give it.
>> >
>> > We can't seem to pin down a reason why we would see this behavior in
>> KVM but
>> > not on metal. I'm hoping someone on the list might have some insight or
>> > ideas.
>> >
>> > Thank You,
>> >
>> > John
>> >
>> > ___
>> > OpenStack-operators mailing list
>> > openstack-operat...@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >
>>
>
>
> ___
> OpenStack-operators mailing list
> openstack-operat...@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] UDP Buffer Filling

[John Petrini]

Hi Liping,

Thank you for the detailed response! I've gone over our environment and
checked the various values.

First I found that we are dropping packets on the physcial nics as well as
inside the instance (though only when its UDP receive buffer overflows).

Our physical nics are using the default ring size and our tap interfaces
are using the 500 tx_txdefault. There are dropped packets on the tap
interfaces but the counts are rather low and don't seem to increase very
often so I'm not sure that there's a problem there but I'm considering
adjusting the value anyway to avoid issues in the future.

We already tune these values in the VM. Would you suggest tuning them on
the compute nodes as well?
net.core.rmem_max / net.core.rmem_default / net.core.wmem_max /
net.core.rmem_default

I'm going to do some testing with multiqueues enabled since both you and
Saverio have suggested it.



John Petrini

Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
*P:* 215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

On Fri, Jul 28, 2017 at 9:25 AM, Erik McCormick <emccorm...@cirrusseven.com>
wrote:

>
>
> On Jul 28, 2017 8:51 AM, "John Petrini" <jpetr...@coredial.com> wrote:
>
> Hi Saverio,
>
> Thanks for the info. The parameter is missing completely:
>
> 
>   
>   
>   
>   
>function='0x0'/>
> 
>
> I've came across the blueprint for adding the image property
> hw_vif_multiqueue_enabled. Do you know if this feature is available in
> Mitaka?
>
> It was merged 2 years ago so should have been there since Liberty.
>
>
> John Petrini
>
> Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
> Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
> <http://www.linkedin.com/company/99631>   [image: Google Plus]
> <https://plus.google.com/104062177220750809525/posts>   [image: Blog]
> <http://success.coredial.com/blog>
>
> 751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
> *P:* 215.297.4400 x232 <(215)%20297-4400>   //   *F: *215.297.4401
> <(215)%20297-4401>   //   *E: *jpetr...@coredial.com
>
>
> On Fri, Jul 28, 2017 at 3:59 AM, Saverio Proto <ziopr...@gmail.com> wrote:
>
>> Hello John,
>>
>> a common problem is packets being dropped when they pass from the
>> hypervisor to the instance. There is bottleneck there.
>>
>> check the 'virsh dumpxml' of one of the instances that is dropping
>> packets. Check for the interface section, should look like:
>>
>> 
>>   
>>   
>>   
>>   
>>   
>>   
>>   > function='0x0'/>
>> 
>>
>> how many queues you have ??? Usually if you have only 1 or if the
>> parameter is missing completely is not good.
>>
>> in Mitaka nova should use 1 queue for every instance CPU core you
>> have. It is worth to check if this is set correctly in your setup.
>>
>> Cheers,
>>
>> Saverio
>>
>>
>>
>> 2017-07-27 17:49 GMT+02:00 John Petrini <jpetr...@coredial.com>:
>> > Hi List,
>> >
>> > We are running Mitaka with VLAN provider networking. We've recently
>> > encountered a problem where the UDP receive queue on instances is
>> filling up
>> > and we begin dropping packets. Moving instances out of OpenStack onto
>> bare
>> > metal resolves the issue completely.
>> >
>> > These instances are running asterisk which should be pulling these
>> packets
>> > off the queue but it appears to be falling behind no matter the
>> resources we
>> > give it.
>> >
>> > We can't seem to pin down a reason why we would see this behavior in
>> KVM but
>> > not on metal. I'm hoping someone on the list might have some insight or
>> > ideas.
>> >
>> > Thank You,
>> >
>> > John
>> >
>> > ___
>> > OpenStack-operators mailing list
>> > OpenStack-operators@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> >
>>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] UDP Buffer Filling

[John Petrini]

Hi Saverio,

Thanks for the info. The parameter is missing completely:


  
  
  
  
  


I've came across the blueprint for adding the image property
hw_vif_multiqueue_enabled.
Do you know if this feature is available in Mitaka?

John Petrini

Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
*P:* 215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

On Fri, Jul 28, 2017 at 3:59 AM, Saverio Proto <ziopr...@gmail.com> wrote:

> Hello John,
>
> a common problem is packets being dropped when they pass from the
> hypervisor to the instance. There is bottleneck there.
>
> check the 'virsh dumpxml' of one of the instances that is dropping
> packets. Check for the interface section, should look like:
>
> 
>   
>   
>   
>   
>   
>   
>function='0x0'/>
> 
>
> how many queues you have ??? Usually if you have only 1 or if the
> parameter is missing completely is not good.
>
> in Mitaka nova should use 1 queue for every instance CPU core you
> have. It is worth to check if this is set correctly in your setup.
>
> Cheers,
>
> Saverio
>
>
>
> 2017-07-27 17:49 GMT+02:00 John Petrini <jpetr...@coredial.com>:
> > Hi List,
> >
> > We are running Mitaka with VLAN provider networking. We've recently
> > encountered a problem where the UDP receive queue on instances is
> filling up
> > and we begin dropping packets. Moving instances out of OpenStack onto
> bare
> > metal resolves the issue completely.
> >
> > These instances are running asterisk which should be pulling these
> packets
> > off the queue but it appears to be falling behind no matter the
> resources we
> > give it.
> >
> > We can't seem to pin down a reason why we would see this behavior in KVM
> but
> > not on metal. I'm hoping someone on the list might have some insight or
> > ideas.
> >
> > Thank You,
> >
> > John
> >
> > ___
> > OpenStack-operators mailing list
> > OpenStack-operators@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack] [Openstack-operators] UDP Buffer Filling

[John Petrini]

Hi Pedro,

Thank you for the suggestion. I will look into this.

John Petrini

Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
*P:* 215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

On Thu, Jul 27, 2017 at 12:25 PM, Pedro Sousa <pgso...@gmail.com> wrote:

> Hi,
>
> have you considered to implement some network acceleration technique like
> to OVS-DPDK or SR-IOV?
>
> In these kind of workloads (voice, video) that have low latency
> requirements you might need to use something like DPDK to avoid these
> issues.
>
> Regards
>
> On Thu, Jul 27, 2017 at 4:49 PM, John Petrini <jpetr...@coredial.com>
> wrote:
>
>> Hi List,
>>
>> We are running Mitaka with VLAN provider networking. We've recently
>> encountered a problem where the UDP receive queue on instances is filling
>> up and we begin dropping packets. Moving instances out of OpenStack onto
>> bare metal resolves the issue completely.
>>
>> These instances are running asterisk which should be pulling these
>> packets off the queue but it appears to be falling behind no matter the
>> resources we give it.
>>
>> We can't seem to pin down a reason why we would see this behavior in KVM
>> but not on metal. I'm hoping someone on the list might have some insight or
>> ideas.
>>
>> Thank You,
>>
>> John
>>
>> ___
>> OpenStack-operators mailing list
>> openstack-operat...@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] UDP Buffer Filling

[John Petrini]

Hi Pedro,

Thank you for the suggestion. I will look into this.

John Petrini

Platforms Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
751 Arbor Way, Hillcrest I, Suite 150, Blue Bell, PA 19422
*P:* 215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

On Thu, Jul 27, 2017 at 12:25 PM, Pedro Sousa <pgso...@gmail.com> wrote:

> Hi,
>
> have you considered to implement some network acceleration technique like
> to OVS-DPDK or SR-IOV?
>
> In these kind of workloads (voice, video) that have low latency
> requirements you might need to use something like DPDK to avoid these
> issues.
>
> Regards
>
> On Thu, Jul 27, 2017 at 4:49 PM, John Petrini <jpetr...@coredial.com>
> wrote:
>
>> Hi List,
>>
>> We are running Mitaka with VLAN provider networking. We've recently
>> encountered a problem where the UDP receive queue on instances is filling
>> up and we begin dropping packets. Moving instances out of OpenStack onto
>> bare metal resolves the issue completely.
>>
>> These instances are running asterisk which should be pulling these
>> packets off the queue but it appears to be falling behind no matter the
>> resources we give it.
>>
>> We can't seem to pin down a reason why we would see this behavior in KVM
>> but not on metal. I'm hoping someone on the list might have some insight or
>> ideas.
>>
>> Thank You,
>>
>> John
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack] UDP Buffer Filling

[John Petrini]

Hi List,

We are running Mitaka with VLAN provider networking. We've recently
encountered a problem where the UDP receive queue on instances is filling
up and we begin dropping packets. Moving instances out of OpenStack onto
bare metal resolves the issue completely.

These instances are running asterisk which should be pulling these packets
off the queue but it appears to be falling behind no matter the resources
we give it.

We can't seem to pin down a reason why we would see this behavior in KVM
but not on metal. I'm hoping someone on the list might have some insight or
ideas.

Thank You,

John
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack-operators] UDP Buffer Filling

[John Petrini]

Hi List,

We are running Mitaka with VLAN provider networking. We've recently
encountered a problem where the UDP receive queue on instances is filling
up and we begin dropping packets. Moving instances out of OpenStack onto
bare metal resolves the issue completely.

These instances are running asterisk which should be pulling these packets
off the queue but it appears to be falling behind no matter the resources
we give it.

We can't seem to pin down a reason why we would see this behavior in KVM
but not on metal. I'm hoping someone on the list might have some insight or
ideas.

Thank You,

John
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack] [Mirantis] How to keep ntpd down

[John Petrini]

To Brad's point - if your controllers are VM's you might also want to have
a look at Chrony https://chrony.tuxfamily.org/. It's supposed to perform
much better on virtual machines.

___

John Petrini

On Thu, Jul 20, 2017 at 9:20 PM, Brad Knowles <b...@shub-internet.org>
wrote:

> On Jul 20, 2017, at 7:26 PM, Raja T Nair <rtn...@gmail.com> wrote:
>
> > Thanks a lot for the reply, John.
> >
> > Yes I understand that time is really important for cluster setup, that's
> why I was panicking and looking for alternatives when I found time drifting
> while ntpd was still on.
> > So I was planning to do a ``ntpdate w.x.y.z '' every 2 mins in order to
> keep time in sync.
> >
> > Would want to investigate this. My upstream time server seems fine, its
> on a baremetal. Many other servers sync with this one too. Also only one
> controller had issues with time.
> > Kind of stuck here, as I have no idea why one node's ntpd would fail :(
>
> Doing a cron job with ntpdate will cause your time to bounce all over the
> place, and that will be even worse than what you've had so far.
>
> I've been a member of the NTP Public Services Project since 2003, and I've
> seen a lot of NTP problems over the years, especially on virtual machines.
> Historically, our advice was to not even run ntpd at all on a VM, but
> instead to run it on the bare hardware underneath, and then make sure that
> you're running the necessary hooks in the hypervisor and the guest OSes to
> pass good quality time up the stack to all the clients.
>
> I'm not sure if that is still the best advice or not -- I think it may
> depend on your hypervisor and your guest OSes.
>
> But if you do run ntpd on the guests, there are things you can do to
> measure larger-than-normal amounts of drift and compensate for that.  I
> would direct you to the mailing list questi...@lists.ntp.org for more
> information.
>
> --
> Brad Knowles <b...@shub-internet.org>
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Mirantis] How to keep ntpd down

[John Petrini]

On all of the controllers? crm resource stop clone_p_ntp should do it.
Although I can't imagine why you would want to do this. Time is very
important in OpenStack (and Ceph if you are running it) which it sounds
like you've already found out.

The whole purpose of NTP is to keep your time in sync - if it's not doing
that you should be looking for the root cause not disabling it. You might
want to start by looking at your upstream time servers that the controllers
are using. This is configured in Fuel and the configuration is stored in
/etc/npt.conf on the controllers.

I'd highly recommend setting up monitoring of ntp so you know when the
clock starts to drift and can respond to it before it drifts too far and
becomes a problem.

___

John Petrini


On Thu, Jul 20, 2017 at 6:29 AM, Raja T Nair <rtn...@gmail.com> wrote:

> Hello All,
>
> Mirantis 7.0
>
> I am trying to keep ntpd down and do a periodic ntpdate against a time
> server.
> This is because one of the controllers started to drift and services in
> that not started to go down.
>
> But it seems that the ntpd daemon comes up after 10 sec every time i stop
> it.
> Is there a monitor running somewhere which does brings it back?
>
> Please guide me on this and also tell me if I am doing something wrong.
>
> Regards,
> Raja.
>
> --
> :^)
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Mirantis Controller has mysql_monitor down

[John Petrini]

Those are failed actions. It only signifies that the monitor failed at some
point not necessarily that there is an ongoing problem. You can clear
failed actions by running crm_resource -P. As long as they don't come back
you should be in good shape.

___

John Petrini

Systems Engineer   //   *CoreDial, LLC*   //   coredial.com   //   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
*P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

[image: Exceptional people. Proven Processes. Innovative Technology.
Discover CoreDial - watch our video]

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Wed, Jul 12, 2017 at 1:16 PM, Raja T Nair <rtn...@gmail.com> wrote:

>
> Hello All,
>
> Mirantis 7.0
> Pacemaker 1.1.12-0u~u14.
>
> My mirantis cluster had time sync issues, and after fixing the same, the
> controller nodes report p_mysql_monitor as Not Running. Can somebody help
> me identify and resolve this issue?
>
> <>
> Failed actions:
> p_mysql_monitor_6 on node-1.mydomain.com 'not running' (7):
> call=138, status=complete, last-rc-change='Wed Jul 12 16:30:55 2017',
> queued=0ms, exec=0ms
> p_mysql_monitor_6 on node-2.mydomain.com 'not running' (7):
> call=138, status=complete, last-rc-change='Wed Jul 12 17:00:47 2017',
> queued=0ms, exec=0ms
> p_mysql_monitor_6 on node-3.mydomain.com 'not running' (7):
> call=154, status=complete, last-rc-change='Wed Jul 12 16:00:28 2017',
> queued=0ms, exec=0ms
>
> 
>
> Regards,
> Raja.
>
> --
> :^)
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] Cannot create provider network

[John Petrini]

How about mysql.


select * from neutron.networks;


On Wed, May 17, 2017 at 11:14 AM, <si...@turka.nl> wrote:

> That is also what I thought, but unfortunately, there is no provider
> network.
>
> The output of neutron net-list is empty.
>
> > It sounds like there's already a provider network. What's the output of
> > neutron net-list?
> >
> > ___
> >
> > John Petrini
> >
> > NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
> > //   [image:
> > Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
> > <http://www.linkedin.com/company/99631>   [image: Google Plus]
> > <https://plus.google.com/104062177220750809525/posts>   [image: Blog]
> > <http://success.coredial.com/blog>
> > Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
> > *P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
> > jpetr...@coredial.com
> >
> >
> > Interested in sponsoring PartnerConnex 2017? Learn more.
> > <http://success.coredial.com/partnerconnex-2017-sponsorship>
> >
> > The information transmitted is intended only for the person or entity to
> > which it is addressed and may contain confidential and/or privileged
> > material. Any review, retransmission,  dissemination or other use of, or
> > taking of any action in reliance upon, this information by persons or
> > entities other than the intended recipient is prohibited. If you received
> > this in error, please contact the sender and delete the material from any
> > computer.
> >
> > On Wed, May 17, 2017 at 11:00 AM, <si...@turka.nl> wrote:
> >
> >> Hello,
> >>
> >> I have followed the installation documentation at
> >> https://docs.openstack.org/newton/install-guide-rdo.
> >>
> >> Everything went fine.
> >>
> >> As stated at
> >> https://docs.openstack.org/newton/install-guide-rdo/
> >> launch-instance-networks-provider.html#launch-instance-
> networks-provider
> >> I tried to create a provider network. I succeeded with this, but I
> >> wanted
> >> to make some changes, so I have deleted it and tried to recreate it, but
> >> it fails with the following:
> >>
> >> [root@controller ~]# openstack network create  --share --external \
> >> >   --provider-physical-network provider \
> >> >   --provider-network-type flat provider
> >> HttpException: Conflict
> >> [root@controller ~]#
> >>
> >> When I try the same with the neutron command, I am getting the
> >> following:
> >>
> >> [root@controller ~]# neutron net-create --shared
> >> --provider:physical_network provider \
> >> >   --provider:network_type flat provider
> >> Unable to create the flat network. Physical network provider is in use.
> >> Neutron server returns request_ids:
> >> ['req-169949d5-56c6-44e8-b5c7-d251dcf98fc2']
> >> [root@controller ~]#
> >>
> >> Any suggestions?
> >>
> >> Thanks!
> >>
> >>
> >> ___
> >> OpenStack-operators mailing list
> >> OpenStack-operators@lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >>
> >
>
>
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack] 答复: [openstack-dev] OpenStack-Ansible HA solution

[John Petrini]

Hi Maxwell,

For compute HA have a look at Masakari -
https://wiki.openstack.org/wiki/Masakari. I don't believe OSA currently
supports it but you could always build your own role for it and the
community would likely be open to integrating it into OSA.


On Thu, Apr 27, 2017 at 5:24 AM, Andy McCrae  wrote:

> Hi Maxwell,
>
>
> I found that OSA use keepalived to resolve controller HA, but whether the
>> keepalived support compute HA? What shoud I do or which file I need to
>> config if I want to resolve compute HA?
>>
>>
>>
>> And there are some solutions for compute HA in this web:
>> http://aspiers.github.io/openstack-summit-2016-austin-compute-ha/ .
>> Would the OSA use pacemaker or some other solutions to support compute HA?
>>
>>
>>
> We don't currently have a nova compute HA solution built-in - although OSA
> itself is quite pluggable, so if you were interested in adding
> functionality to OSA for that - by utilizing the inventory and roles
> that are setup in OSA you would be able to achieve this.
>
> However, as it stands there is only functionality for adding keepalived HA
> functionality for the controller/API services.
>
> Andy
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [openstack-dev] [Openstack] 答复: OpenStack-Ansible HA solution

[John Petrini]

Hi Maxwell,

For compute HA have a look at Masakari -
https://wiki.openstack.org/wiki/Masakari. I don't believe OSA currently
supports it but you could always build your own role for it and the
community would likely be open to integrating it into OSA.


On Thu, Apr 27, 2017 at 5:24 AM, Andy McCrae  wrote:

> Hi Maxwell,
>
>
> I found that OSA use keepalived to resolve controller HA, but whether the
>> keepalived support compute HA? What shoud I do or which file I need to
>> config if I want to resolve compute HA?
>>
>>
>>
>> And there are some solutions for compute HA in this web:
>> http://aspiers.github.io/openstack-summit-2016-austin-compute-ha/ .
>> Would the OSA use pacemaker or some other solutions to support compute HA?
>>
>>
>>
> We don't currently have a nova compute HA solution built-in - although OSA
> itself is quite pluggable, so if you were interested in adding
> functionality to OSA for that - by utilizing the inventory and roles
> that are setup in OSA you would be able to achieve this.
>
> However, as it stands there is only functionality for adding keepalived HA
> functionality for the controller/API services.
>
> Andy
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openst...@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [Openstack] Static IP for VM

[John Petrini]

Hi Georgios,

Yes you can. You would need to create a new port in neutron using the
--fixed-ip flag to assign the IP you want. Then attach the port to you
instances and DHCP will hand it that IP.

___

John Petrini

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] [Openstack] Slow internet speed - Instances with floating IP

[John Petrini]

Hi All,

Turns out this was a DNS issue. The speedtest-cli script does a lot of
lookups and these lookups were hanging due to parallel A and  requests
resulting in the unexpected results. I added the following to
/etc/resolv.conf and the issue seems to be resolved.

options single-request-reopen

Thanks,

___

John Petrini
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack] Slow internet speed - Instances with floating IP

[John Petrini]

Hi Mohit,

The tenant networks use an MTU of 8950. The floating network uses and MTU
of 9000.

neutron net-show 4305e3d5-b083-4f33-a2a0-1dfccf1238b6
+---+--+
| Field | Value|
+---+--+
| admin_state_up| True |
| availability_zone_hints   |  |
| availability_zones| nova |
| created_at| 2016-10-10T15:35:44  |
| description   |  |
| id| 4305e3d5-b083-4f33-a2a0-1dfccf1238b6 |
| ipv4_address_scope|  |
| ipv6_address_scope|  |
| mtu   | 8950 |
| name  | systems-net  |
| port_security_enabled | True |
| provider:network_type | vxlan|
| provider:physical_network |  |
| provider:segmentation_id  | 100  |
| router:external   | False|
| shared| False|
| status| ACTIVE   |
| subnets   | d1265c93-d47c-4902-89a6-8ad3172fb946 |
| tags  |  |
| tenant_id | c7c31d0bd344476a8c8ca39a19e1be15 |
| updated_at| 2016-10-10T15:35:44  |
+---+--+

neutron net-show 4c7544f3-c06b-48cf-babc-015d29dd7e62
+---+--+
| Field | Value|
+---+--+
| admin_state_up| True |
| availability_zone_hints   |  |
| availability_zones| nova |
| created_at| 2016-09-23T11:07:56  |
| description   |  |
| id| 4c7544f3-c06b-48cf-babc-015d29dd7e62 |
| ipv4_address_scope|  |
| ipv6_address_scope|  |
| is_default| False|
| mtu   | 9000 |
| name  | admin_floating_net   |
| port_security_enabled | True |
| provider:network_type | flat |
| provider:physical_network | physnet1 |
| provider:segmentation_id  |  |
| router:external   | True |
| shared| False|
| status| ACTIVE   |
| subnets   | c70635c4-9bf7-4e4d-9353-c619e196cdf0 |
| tags  |  |
| tenant_id | a2b6f1b348e54229b3244183d6aa41a5 |
| updated_at| 2016-09-23T11:07:56  |
+---+--+

Thank You,

___

John Petrini
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Slow internet speed - Instances with floating IP

[John Petrini]

Adding the OpenStack list in hopes this might get some attention there.

Thanks All,

___

John Petrini
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack-operators] Slow internet speed - Instances with floating IP

[John Petrini]

Adding the OpenStack list in hopes this might get some attention there.

Thanks All,

___

John Petrini
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack] Neutron Network DNS Suffix via DHCP

[John Petrini]

I'm also interested in this.

Thank You,

John Petrini
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Reg. NAT

[John Petrini]

Hi Vikram,

You may want to look into provider networks. Here's some documentation for
doing so in an Open vSwitch deployment.

https://docs.openstack.org/liberty/networking-guide/scenario-provider-ovs.html

Provider networks allow you to add existing networks outside of OpenStack
to your cloud. Instances can be attached to these networks directly
therefore bypassing the NAT that's required for floating IP's.

These networks do not require a virtual router (warning: you can still
create them!) as they rely on existing routing outside of OpenStack. For
this reason they can only be created by an admin and you should also
consider keeping them private and granting access to tenants using RBAC.

We use them extensively and they work well. We're able to add new networks
as needed by trunking the VLAN of the desired network to the compute and
controller nodes and then creating the new provider type network in
neutron. Providing a segmentation id when creating the network allows Open
vSwitch to tag the traffic with the proper vlan before it leaves the
compute node.

Regards,

John Petrini
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Snapshot: Cannot determine the parent storage pool

[John Petrini]

That's a great question and I'm honestly not sure. Anytime I've tried to do
so myself I've also received an error. A colleague of mine first renamed
the image and then deleted it without issue. I'm going to see if I can
reproduce this somehow. Thank you for the link - I may end up going a
simpler route and just snapshooting the affected instances and rebuilding
them from the snapshots so they have a base image again.

Here's the nova output from an affected VM:

nova show 63ef9e03-0b0e-4d89-9334-3a68af0a2731 | grep image
| image| Image not found
(cfa4f24e-24f5-4c03-8edf-8d0469247fa5)
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack] Snapshot: Cannot determine the parent storage pool

[John Petrini]

Hi List,

We're running Mitaka with Ceph. Recently I enabled RBD snapshots by adding
write permissions to the images pool in Ceph. This works perfectly for some
instances but is failing back to standard snapshots for others with the
following error:

Performing standard snapshot because direct snapshot failed: Cannot
determine the parent storage pool for 7a7b5119-
85da-429b-89b5-ad345cfb649e; cannot determine where to store images

Looking at the code here:
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/imagebackend.py
it appears that it looks for the pool of the base image to determine where
to save the snapshot. I believe the problem I'm encountering is that for
some of our instances the base image no longer exists.

Am I understanding this correctly and is there anyway to explicitly set the
pool to be used for snapshots and bypass this logic?

Thank You,

John Petrini
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Fuel] Use other NIC

[John Petrini]

I've always run it as a VM. In my opinion there's no real need to dedicate
hardware to Fuel since it doesn't even need to be running all the time.
Once you deploy your cloud you can shut it down until the next time it's
needed.

That will solve you eth0 problem also.

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Sun, Jan 29, 2017 at 1:12 PM, Georgios Dimitrakakis  wrote:

> Hi!
>
> I am a new user of Fuel and would like to deploy it on a physical
> SuperMicro machine. Unfortunately I wasn't able to install it via the ISO
> file since I 've tried several times (with both the community and the
> Mirantis edition) and every time there is a problem with the boot loader
> which cannot be installed correctly thus resulting in a non-bootable
> system. That's the reason I moved on, I installed CentOS and tried to
> follow the instructions from here: http://docs.openstack.org/deve
> loper/fuel-docs/userdocs/fuel-install-guide/install/install_
> install_fuel_master_node.html
>
> Even this attempt was without success since the Fuel bootstrap script is
> looking for connectivity on eth0 which is not connected.
>
> The machine is the property of a DataCenter and out of the 4interfaces
> only the two are cabled and those two are not the first ones but the third
> and forth. So my CentOS has connectivity on "eno3" and "eno4" devices. So
> my question is how can I instruct Fuel to look for connectivity on "eno3"
> rather than on "eth0" or "eno1"?
>
> I did try to disable the first two interfaces from the BIOS but couldn't
> find any option. Then I thought that I could perform a trick and rename the
> devices but that doesn't seem correct...If there isn't any other option I
> will try it but I am sure that Fuel has a way to instract the bootstarp
> script to use a specific device but haven't found it yet.
>
> Your help is much appreciated!
>
> Thank you,
>
>
> George
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] nova backup - instances unreachable

[John Petrini]

Adding disable_libvirt_livesnapshot = false to the nova.conf enabled live
snapshots.

Thanks everyone for the help.

___

John Petrini

NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
//   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
*P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

[image: Exceptional people. Proven Processes. Innovative Technology.
Discover CoreDial - watch our video]
<http://cta-redirect.hubspot.com/cta/redirect/210539/4c492538-6e4b-445e-9480-bef676787085>

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Mon, Jan 23, 2017 at 3:55 AM, John Petrini <jpetr...@coredial.com> wrote:

> Hi Eugen,
>
> disable_libvirt_livesnapshot is not present in the nova.conf. During the
> snapshot the nova logs says "Beginning cold snapshot".
>
> I read about this option in the nova documentation but did not realize it
> was the default. In fact I assumed it wasn't since it's in the workarounds
> section. I'll try setting it to false.
>
> Thank You,
>
>
>
> ___
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission,  dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
> On Mon, Jan 23, 2017 at 3:16 AM, Eugen Block <ebl...@nde.ag> wrote:
>
>> Have you enabled live snapshots in nova.conf?
>>
>> The default for this option is "true", so you should check that:
>>
>> disable_libvirt_livesnapshot = false
>>
>> Is it really a live snaphot? What's in the nova-compute.log? It should
>> say something like
>>
>> [instance: XXX] Beginning live snapshot process
>>
>>
>>
>> Regards,
>> Eugen
>>
>>
>>
>> Zitat von John Petrini <jpetr...@coredial.com>:
>>
>> Hi All,
>>>
>>> Following up after making this change. Adding write permissions to the
>>> images pool in Ceph did the trick and RBD snapshots now work. However the
>>> instance is still paused for the duration of the snapshot. Is it possible
>>> to do a live snapshot without pausing the instance?
>>>
>>> Thanks,
>>>
>>> John
>>>
>>> On Fri, Jan 13, 2017 at 5:49 AM, Eugen Block <ebl...@nde.ag> wrote:
>>>
>>> Thanks,
>>>>
>>>> for anyone interested in this issue, I filed a bug report:
>>>> https://bugs.launchpad.net/nova/+bug/1656242
>>>>
>>>>
>>>> Regards,
>>>> Eugen
>>>>
>>>>
>>>> Zitat von Mohammed Naser <mna...@vexxhost.com>:
>>>>
>>>> It is likely because this has been tested with QEMU only. I think you
>>>>
>>>>> might want to bring this up with the Nova team.
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> On Jan 12, 2017, at 11:28 AM, Eugen Block <ebl...@nde.ag> wrote:
>>>>>
>>>>>>
>>>>>> I'm not sure if this is the right spot, but I added some log
>>>>>> statements
>>>>>> into driver.py.
>>>>>> First, there's this if-block:
>>>>>>
>>>>>>if (self._host.has_min_version(MI
>>>>>> N_LIBVIRT_LIVESNAPSHOT_VERSION,
>>>>>>   MIN_QEMU_LIVESNAPSHOT_VERSION,
>>>>>>   host.HV_DRIVER_QEMU)
>>>>>> and source_type not in ('lvm')
>>>>>> and not CONF.ephemeral_storage_encryption.enabled
>>>>>> and not CONF.workarounds.disable_libvirt_livesnapshot):
>&g

Re: [Openstack] nova backup - instances unreachable

[John Petrini]

Hi Eugen,

disable_libvirt_livesnapshot is not present in the nova.conf. During the
snapshot the nova logs says "Beginning cold snapshot".

I read about this option in the nova documentation but did not realize it
was the default. In fact I assumed it wasn't since it's in the workarounds
section. I'll try setting it to false.

Thank You,



___

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Mon, Jan 23, 2017 at 3:16 AM, Eugen Block <ebl...@nde.ag> wrote:

> Have you enabled live snapshots in nova.conf?
>
> The default for this option is "true", so you should check that:
>
> disable_libvirt_livesnapshot = false
>
> Is it really a live snaphot? What's in the nova-compute.log? It should say
> something like
>
> [instance: XXX] Beginning live snapshot process
>
>
>
> Regards,
> Eugen
>
>
>
> Zitat von John Petrini <jpetr...@coredial.com>:
>
> Hi All,
>>
>> Following up after making this change. Adding write permissions to the
>> images pool in Ceph did the trick and RBD snapshots now work. However the
>> instance is still paused for the duration of the snapshot. Is it possible
>> to do a live snapshot without pausing the instance?
>>
>> Thanks,
>>
>> John
>>
>> On Fri, Jan 13, 2017 at 5:49 AM, Eugen Block <ebl...@nde.ag> wrote:
>>
>> Thanks,
>>>
>>> for anyone interested in this issue, I filed a bug report:
>>> https://bugs.launchpad.net/nova/+bug/1656242
>>>
>>>
>>> Regards,
>>> Eugen
>>>
>>>
>>> Zitat von Mohammed Naser <mna...@vexxhost.com>:
>>>
>>> It is likely because this has been tested with QEMU only. I think you
>>>
>>>> might want to bring this up with the Nova team.
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Jan 12, 2017, at 11:28 AM, Eugen Block <ebl...@nde.ag> wrote:
>>>>
>>>>>
>>>>> I'm not sure if this is the right spot, but I added some log statements
>>>>> into driver.py.
>>>>> First, there's this if-block:
>>>>>
>>>>>if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION
>>>>> ,
>>>>>   MIN_QEMU_LIVESNAPSHOT_VERSION,
>>>>>   host.HV_DRIVER_QEMU)
>>>>> and source_type not in ('lvm')
>>>>> and not CONF.ephemeral_storage_encryption.enabled
>>>>> and not CONF.workarounds.disable_libvirt_livesnapshot):
>>>>>live_snapshot = True
>>>>>   [...]
>>>>>else:
>>>>>live_snapshot = False
>>>>>
>>>>> And I know that it lands in the else-statement. Turns out that
>>>>> _host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We
>>>>> are
>>>>> running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN and
>>>>> now
>>>>> nova-compute says:
>>>>>
>>>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
>>>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live
>>>>> snapshot process
>>>>>
>>>>> Now I'm waiting for the result, but at least the VM is still running,
>>>>> so
>>>>> it looks quite promising...
>>>>>
>>>>> And there it is:
>>>>>
>>>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image upload
>>>>> complete
>>>>>
>>>>> I'm testing the image now, and it works!
>>>>>
>>>>> Now the question is, why is it defaulting to HV_DRIVER_QEMU and is it
>>>>> really necessary to change this directly in the code? Is there any
>>>>> other
>>>>> way?
>>>>>
>>>>> Regards,
>>>>> Eugen
>>>>>
>>>>> Zitat von Eugen Block <ebl...@nde.ag>:
>>>>>
>>>>> Yes, I truncated the

Re: [Openstack] [OpenStack] VM start up with no route rules

[John Petrini]

In my environment they are both Status = Active Admin = Up

___

John Petrini

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Sat, Jan 21, 2017 at 9:31 PM, Xu, Rongjie (Nokia - CN/Hangzhou) <
rongjie...@nokia.com> wrote:

> OK.
>
>
>
> But how about the status? Per my understanding, one of them should be
> “STATUS=ACTIVE, ADMIN STATUS=UP” and the other should be “STATUS=DOWN,
> ADMIN STATUS=UP”. Otherwise, it should be wrong. Right?
>
>
>
> Best Regards
>
> Xu Rongjie (Max)
>
> Mobile:18658176819 <(865)%20817-6819>
>
>
>
> *From:* John Petrini [mailto:jpetr...@coredial.com]
> *Sent:* Sunday, January 22, 2017 10:19
> *To:* Xu, Rongjie (Nokia - CN/Hangzhou) <rongjie...@nokia.com>
> *Cc:* Eugen Block <ebl...@nde.ag>; openstack@lists.openstack.org
>
> *Subject:* Re: [Openstack] [OpenStack] VM start up with no route rules
>
>
>
> Multiple DHCP ports in not uncommon in HA configurations. You can check if
> this is enabled in /etc/neutron/neutron.conf.
>
>
>
> dhcp_agents_per_network = 2
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission,  dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
>
>
> On Sat, Jan 21, 2017 at 8:13 PM, Xu, Rongjie (Nokia - CN/Hangzhou) <
> rongjie...@nokia.com> wrote:
>
> Hi,
>
> Thanks for your analysis. Finally, I found there are two DHCP ports when I
> create a tenant network with DHCP enabled. That makes VM cannot start up
> due to fail to get its ip address.
>
> But why there two DHCP ports when I create the network is still under
> investigation. Do you have some hints for this? Thanks.
>
> Best Regards
> Xu Rongjie (Max)
> Mobile:18658176819
>
> -Original Message-
> From: Eugen Block [mailto:ebl...@nde.ag]
> Sent: Thursday, January 19, 2017 23:57
> To: openstack@lists.openstack.org
> Subject: Re: [Openstack] [OpenStack] VM start up with no route rules
>
> Does your VM's interface also have DHCP enabled? If it's configured to
> have a static address, it won't be changed by dhcp. Have you used the
> image outside of heat and did it work with dhcp for a single VM?
>
>
> Zitat von "Xu, Rongjie (Nokia - CN/Hangzhou)" <rongjie...@nokia.com>:
>
> > Hi,
> >
> > I am launch a heat stack on top of Mirantis OpenStack Mitaka.
> > However, I see no route rules (output of command 'ip route' is
> > empty) inside VM, which make the VM cannot get the metadata from
> > metadata server. Basically, the VM is connected to a management
> > network (192.168.1.0/24 DHCP enabled).
> >
> > How can I debug this problem? Is it something wrong with Neutron? Thanks.
> >
> >
> >
> > Best Regards
> > Xu Rongjie (Max)
> > Mobile:18658176819
>
>
>
> --
> Eugen Block voice   : +49-40-559 51 75
> NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
> Postfach 61 03 15
> D-22423 Hamburg e-mail  : ebl...@nde.ag
>
>  Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>Sitz und Registergericht: Hamburg, HRB 90934
>Vorstand: Jens-U. Mozdzen
> USt-IdNr. DE 814 013 983
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [OpenStack] VM start up with no route rules

[John Petrini]

Multiple DHCP ports in not uncommon in HA configurations. You can check if
this is enabled in /etc/neutron/neutron.conf.

dhcp_agents_per_network = 2

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Sat, Jan 21, 2017 at 8:13 PM, Xu, Rongjie (Nokia - CN/Hangzhou) <
rongjie...@nokia.com> wrote:

> Hi,
>
> Thanks for your analysis. Finally, I found there are two DHCP ports when I
> create a tenant network with DHCP enabled. That makes VM cannot start up
> due to fail to get its ip address.
>
> But why there two DHCP ports when I create the network is still under
> investigation. Do you have some hints for this? Thanks.
>
> Best Regards
> Xu Rongjie (Max)
> Mobile:18658176819
>
>
> -Original Message-
> From: Eugen Block [mailto:ebl...@nde.ag]
> Sent: Thursday, January 19, 2017 23:57
> To: openstack@lists.openstack.org
> Subject: Re: [Openstack] [OpenStack] VM start up with no route rules
>
> Does your VM's interface also have DHCP enabled? If it's configured to
> have a static address, it won't be changed by dhcp. Have you used the
> image outside of heat and did it work with dhcp for a single VM?
>
>
> Zitat von "Xu, Rongjie (Nokia - CN/Hangzhou)" :
>
> > Hi,
> >
> > I am launch a heat stack on top of Mirantis OpenStack Mitaka.
> > However, I see no route rules (output of command 'ip route' is
> > empty) inside VM, which make the VM cannot get the metadata from
> > metadata server. Basically, the VM is connected to a management
> > network (192.168.1.0/24 DHCP enabled).
> >
> > How can I debug this problem? Is it something wrong with Neutron? Thanks.
> >
> >
> >
> > Best Regards
> > Xu Rongjie (Max)
> > Mobile:18658176819
>
>
>
> --
> Eugen Block voice   : +49-40-559 51 75
> NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
> Postfach 61 03 15
> D-22423 Hamburg e-mail  : ebl...@nde.ag
>
>  Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>Sitz und Registergericht: Hamburg, HRB 90934
>Vorstand: Jens-U. Mozdzen
> USt-IdNr. DE 814 013 983
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] nova backup - instances unreachable

[John Petrini]

Hi All,

Following up after making this change. Adding write permissions to the
images pool in Ceph did the trick and RBD snapshots now work. However the
instance is still paused for the duration of the snapshot. Is it possible
to do a live snapshot without pausing the instance?

Thanks,

John

On Fri, Jan 13, 2017 at 5:49 AM, Eugen Block <ebl...@nde.ag> wrote:

> Thanks,
>
> for anyone interested in this issue, I filed a bug report:
> https://bugs.launchpad.net/nova/+bug/1656242
>
>
> Regards,
> Eugen
>
>
> Zitat von Mohammed Naser <mna...@vexxhost.com>:
>
> It is likely because this has been tested with QEMU only. I think you
>> might want to bring this up with the Nova team.
>>
>> Sent from my iPhone
>>
>> On Jan 12, 2017, at 11:28 AM, Eugen Block <ebl...@nde.ag> wrote:
>>>
>>> I'm not sure if this is the right spot, but I added some log statements
>>> into driver.py.
>>> First, there's this if-block:
>>>
>>>if (self._host.has_min_version(MIN_LIBVIRT_LIVESNAPSHOT_VERSION,
>>>   MIN_QEMU_LIVESNAPSHOT_VERSION,
>>>   host.HV_DRIVER_QEMU)
>>> and source_type not in ('lvm')
>>> and not CONF.ephemeral_storage_encryption.enabled
>>> and not CONF.workarounds.disable_libvirt_livesnapshot):
>>>live_snapshot = True
>>>   [...]
>>>else:
>>>live_snapshot = False
>>>
>>> And I know that it lands in the else-statement. Turns out that
>>> _host.has_min_version is "false", because of host.HV_DRIVER_QEMU. We are
>>> running on Xen hypervisors. So I tried it with host.HV_DRIVER_XEN and now
>>> nova-compute says:
>>>
>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] instance snapshotting
>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Beginning live
>>> snapshot process
>>>
>>> Now I'm waiting for the result, but at least the VM is still running, so
>>> it looks quite promising...
>>>
>>> And there it is:
>>>
>>> [instance: 14b75237-7619-481f-9636-792b64d1be17] Snapshot image upload
>>> complete
>>>
>>> I'm testing the image now, and it works!
>>>
>>> Now the question is, why is it defaulting to HV_DRIVER_QEMU and is it
>>> really necessary to change this directly in the code? Is there any other
>>> way?
>>>
>>> Regards,
>>> Eugen
>>>
>>> Zitat von Eugen Block <ebl...@nde.ag>:
>>>
>>> Yes, I truncated the file and uploaded it:
>>>>
>>>> http://dropcanvas.com/ta7nu
>>>> (First time I used this service, please give me feedback if this
>>>> doesn't work for you)
>>>>
>>>> I see the "Beginning cold snapshot process" message, but I don't know
>>>> why. Any help is appreciated!
>>>>
>>>> Regards,
>>>> Eugen
>>>>
>>>>
>>>> Zitat von Mohammed Naser <mna...@vexxhost.com>:
>>>>
>>>> Would you be able to share the logs of a full snapshot run with the
>>>>> compute node in debug?
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> On Jan 12, 2017, at 7:47 AM, Eugen Block <ebl...@nde.ag> wrote:
>>>>>>
>>>>>> That's strange, I also searched for this message, but nothing there.
>>>>>> I have debug logs enabled on compute node but I don't see anything
>>>>>> regarding ceph. No matter, what I do, my instance is always shutdown 
>>>>>> before
>>>>>> a snapshot is taken. What else can I try?
>>>>>>
>>>>>>
>>>>>> Zitat von John Petrini <jpetr...@coredial.com>:
>>>>>>
>>>>>> Mohammed,
>>>>>>>
>>>>>>> It looks like you may be right. Just found the permissions issue in
>>>>>>> the
>>>>>>> nova log on the compute node.
>>>>>>>
>>>>>>> 4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
>>>>>>> 49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
>>>>>>> because direct snapshot failed: no write permission on storage pool
>>>>>>> images
>>>>>>>
>>>>>>> I'm going to 

Re: [Openstack] nova backup - instances unreachable

[John Petrini]

Mohammed,

It looks like you may be right. Just found the permissions issue in the
nova log on the compute node.

4-e8f52e4fbcfb 691caf1c10354efab3e3c8ed61b7d89a
49bc5e5bf2684bd0948d9f94c7875027 - - -] Performing standard snapshot
because direct snapshot failed: no write permission on storage pool images

I'm going to test the change and will send an update you all with the
results.

Thank You,

___

John Petrini



>>
> Yes, we are also running Mitaka and I also read Sebastien Han's blogs ;-)
>
> our snapshots are not happening at the RBD level,
>> they are being copied and uploaded to glance which takes up a lot of space
>> and is very slow.
>>
>
> Unfortunately, that's what we are experiencing, too. I don't know if
> there's something I missed in the nova configs or somewhere else, but I'm
> relieved that I'm not the only one :-)
>
> While writing this email I searched again and found something:
>
> https://specs.openstack.org/openstack/nova-specs/specs/mitak
> a/implemented/rbd-instance-snapshots.html
>
> https://review.openstack.org/#/c/205282/
>
> It seems to be implemented already, I'm looking for the config options to
> set. If you manage to get nova to make rbd snapshots, please let me know ;-)
>
> Regards,
> Eugen
>
>
>
> Zitat von John Petrini <jpetr...@coredial.com>:
>
> Hi Eugen,
>>
>> Thanks for the response! That makes a lost of sense and is what I figured
>> was going on but I missed it in the documentation. We use Ceph as well and
>> I had considered doing the snapshots at the RBD level but I was hoping
>> there was someway to accomplish this via nova. I came across this
>> Sebastien
>> Han write-up that claims this functionality was added to Mitaka:
>> http://www.sebastien-han.fr/blog/2015/10/05/openstack-nova-
>> snapshots-on-ceph-rbd/
>>
>> We are running Mitaka but our snapshots are not happening at the RBD
>> level,
>> they are being copied and uploaded to glance which takes up a lot of space
>> and is very slow.
>>
>> Have you or anyone else implemented this in Mitaka? Other than Sebastian's
>> blog I haven't found any documentation on this.
>>
>> Thank You,
>>
>> ___
>>
>> John Petrini
>>
>> On Wed, Jan 11, 2017 at 3:32 AM, Eugen Block <ebl...@nde.ag> wrote:
>>
>> Hi,
>>>
>>> this seems to be exptected, the docs say:
>>>
>>> "Shut down the source VM before you take the snapshot to ensure that all
>>> data is flushed to disk."
>>>
>>> So if the VM is not shut down, it's freezed to prevent data loss (I
>>> guess). Depending on your storage backend, there are other ways to
>>> perform
>>> backups of your VMs.
>>> We use Ceph as backend for nova, glance and cinder. Ceph stores the
>>> disks,
>>> images and volumes as Rados block device objects. We have a backup script
>>> that creates snapshots of these RBDs, which are exported to our backup
>>> drive. This way the running VM is not stopped or freezed, the user
>>> doesn't
>>> notice any issues. Unlike a nova snapshot, the rbd snapshot is created
>>> immediately within a few seconds. After a successful backup the snapshots
>>> are removed.
>>>
>>> Hope this helps! If you are interested in Ceph, visit [1].
>>>
>>> Regards,
>>> Eugen
>>>
>>> [1] http://docs.ceph.com/docs/giant/start/intro/
>>>
>>>
>>> Zitat von John Petrini <jpetr...@coredial.com>:
>>>
>>>
>>> Hello,
>>>
>>>>
>>>> I've just started experimenting with nova backup and discovered that
>>>> there
>>>> is a period of time during the snapshot where the instance becomes
>>>> unreachable. Is this behavior expected during a live snapshot? Is there
>>>> any
>>>> way to prevent this?
>>>>
>>>> ___
>>>>
>>>> John Petrini
>>>>
>>>>
>>>
>>>
>>> --
>>> Eugen Block voice   : +49-40-559 51 75
>>> NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
>>> Postfach 61 03 15
>>> D-22423 Hamburg e-mail  : ebl...@nde.ag
>>>
>>> Vorsitzende des Aufsichtsrates: Angelika Mozdzen
>>>   Sitz und Registergericht: Hamburg, HRB 90934
>>>   Vorstand: Jens-U. Mozdzen
>>>USt-IdNr. DE 814 013 983
>>>
>>>
>>> __

[Openstack] nova backup - instances unreachable

[John Petrini]

Hello,

I've just started experimenting with nova backup and discovered that there
is a period of time during the snapshot where the instance becomes
unreachable. Is this behavior expected during a live snapshot? Is there any
way to prevent this?

___

John Petrini
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [Neutron][VXLAN] PPTP VPN problem

[John Petrini]

Are you using OpenVPN? In that case I believe you need the following
options on the client:

mssfix 1200
tun-mtu 1200

___

John Petrini

NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
//   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
*P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

[image: Exceptional people. Proven Processes. Innovative Technology.
Discover CoreDial - watch our video]
<http://cta-redirect.hubspot.com/cta/redirect/210539/4c492538-6e4b-445e-9480-bef676787085>

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Thu, Jan 5, 2017 at 11:48 AM, Mārtiņš Jakubovičs <
martins-li...@hostnet.lv> wrote:

> I am tying, but all the time VPN starts with 1500 MTU.
>
> On 2017.01.05. 18:46, John Petrini wrote:
>
> Is ppp0 your VPN interface? Have you tried lowering thee MTU on the VPN?
>
> ___
>
> John Petrini
>
> NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
>//   [image: Twitter] <https://twitter.com/coredial>   [image:
> LinkedIn] <http://www.linkedin.com/company/99631>   [image: Google Plus]
> <https://plus.google.com/104062177220750809525/posts>   [image: Blog]
> <http://success.coredial.com/blog>
> Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
> *P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
> <jpetr...@coredial.com>jpetr...@coredial.com
>
> [image: Exceptional people. Proven Processes. Innovative Technology.
> Discover CoreDial - watch our video]
> <http://cta-redirect.hubspot.com/cta/redirect/210539/4c492538-6e4b-445e-9480-bef676787085>
>
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission,  dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
> On Thu, Jan 5, 2017 at 11:20 AM, Mārtiņš Jakubovičs <
> <martins-li...@hostnet.lv>martins-li...@hostnet.lv> wrote:
>
>> Network it self are working great, but I just have issue with PPTP.
>>
>> If I lower MTU in instance interface, it didn't help.
>> On 2017.01.05. 18:11, Prashant Shetty wrote:
>>
>> Could you try setting MTU lesser value say 1200 on instance interface and
>> re-try?.
>> Ideally your physical server connecting to  switch port must have
>> configured with atleast 1600 MTU in order for VXLAN traffic to work
>> seamlessly.
>>
>> Thanks,
>> Prashant
>>
>> On Thu, Jan 5, 2017 at 9:30 PM, Mārtiņš Jakubovičs <
>> <martins-li...@hostnet.lv>martins-li...@hostnet.lv> wrote:
>>
>>> Hello all,
>>>
>>> I have problem to setup PPTP VPN behind floating IP's in instances which
>>> have VXLAN network. Maybe someone had same issue and can recommend
>>> something? In security groups I allowed protocol's 47 traffic and VPN
>>> client can connect but it can't establish connection, I consider, maybe
>>> issue are with MTU, so VXLAN have MTU 1450, but if I watch ip link show, I
>>> can see that ppp0 interface creates with MTU of 1500.
>>>
>>> Best regards,
>>>
>>> Martins
>>>
>>>
>>> ___
>>> Mailing list: http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>> Post to : openstack@lists.openstack.org
>>> Unsubscribe : http://lists.openstack.org/cgi
>>> -bin/mailman/listinfo/openstack
>>>
>>
>>
>>
>> ___
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] To make compute nodes redundant

[John Petrini]

I highly recommend Masakari. You can find it here:
https://github.com/openstack/masakari


We're using the original version found here:
https://github.com/ntt-sic/masakari but I believe they've officially moved
to the OpenStack version now and are recommending that over the original.



On Wed, Dec 28, 2016 at 2:49 AM, Shinobu Kinjo  wrote:

> You may be interested in instance ha using pacemaker + pacemaker remote.
>
> Regards,
>
> On Wed, Dec 28, 2016 at 4:28 PM, Atif Munir  wrote:
> > Hi,
> >
> > I have an OpenStack installation with
> >
> > 1 x controller node
> > 3 x compute nodes
> >
> > How I can declare one compute as redundant, I am afraid if something goes
> > wrong with one compute, how the Instances are going to survive. Thanks
> >
> > Atif
> >
> > ___
> > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> > Post to : openstack@lists.openstack.org
> > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> >
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Max open files limit for nova-api

[John Petrini]

Hi Prashant,

On second thought that trick might only work on CentOS.  You might have
success using prlimit instead.

___

John Petrini

NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
//   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
*P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

[image: Exceptional people. Proven Processes. Innovative Technology.
Discover CoreDial - watch our video]
<http://cta-redirect.hubspot.com/cta/redirect/210539/4c492538-6e4b-445e-9480-bef676787085>

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Mon, Dec 19, 2016 at 1:13 PM, Prashant Shetty <
prashantshetty1...@gmail.com> wrote:

> Hi John,
>
> Echo option doesnt seems to work on below controller.
>
> stack@devstackvm:~$ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=14.04
> DISTRIB_CODENAME=trusty
> DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
> stack@devstackvm:~$
>
> stack@devstackvm:~$ ps aux | grep nova-api
> stack 3070  1.1  0.1 271576 121092 pts/9   S+   Dec16  50:29
> /usr/bin/python /usr/local/bin/nova-api
> stack 3079  0.3  0.8 1045308 881676 pts/9  S+   Dec16  14:32
> /usr/bin/python /usr/local/bin/nova-api
> stack 3080  0.2  0.1 324808 161096 pts/9   S+   Dec16   9:25
> /usr/bin/python /usr/local/bin/nova-api
> stack 3081  0.2  0.7 980172 816468 pts/9   S+   Dec16  10:37
> /usr/bin/python /usr/local/bin/nova-api
> stack 3082  0.3  0.1 336824 173288 pts/9   S+   Dec16  16:11
> /usr/bin/python /usr/local/bin/nova-api
> stack 3083  0.4  0.1 338788 175264 pts/9   S+   Dec16  19:05
> /usr/bin/python /usr/local/bin/nova-api
> stack 3084  0.4  0.1 336616 172912 pts/9   S+   Dec16  17:41
> /usr/bin/python /usr/local/bin/nova-api
> stack 3085  0.2  0.8 1054900 891272 pts/9  S+   Dec16  10:09
> /usr/bin/python /usr/local/bin/nova-api
> stack 3086  0.2  0.1 325088 161228 pts/9   S+   Dec16   9:59
> /usr/bin/python /usr/local/bin/nova-api
> stack 3097  0.2  0.1 308088 151828 pts/9   S+   Dec16  11:10
> /usr/bin/python /usr/local/bin/nova-api
> stack 3098  0.2  0.1 308296 152360 pts/9   S+   Dec16  11:15
> /usr/bin/python /usr/local/bin/nova-api
> stack 3099  0.2  0.1 308708 152544 pts/9   S+   Dec16  11:42
> /usr/bin/python /usr/local/bin/nova-api
> stack 3100  0.2  0.1 309588 153624 pts/9   S+   Dec16  11:06
> /usr/bin/python /usr/local/bin/nova-api
> stack 3101  0.2  0.1 308372 152396 pts/9   S+   Dec16  11:14
> /usr/bin/python /usr/local/bin/nova-api
> stack 3102  0.2  0.1 308084 152052 pts/9   S+   Dec16  11:10
> /usr/bin/python /usr/local/bin/nova-api
> stack 3103  0.2  0.1 308380 152416 pts/9   S+   Dec16  11:09
> /usr/bin/python /usr/local/bin/nova-api
> stack 3104  0.2  0.1 307652 151560 pts/9   S+   Dec16  10:29
> /usr/bin/python /usr/local/bin/nova-api
> stack 8087  0.0  0.0  11752  2044 pts/21   S+   10:07   0:00 grep
> --color=auto nova-api
> stack@devstackvm:~$
>
> stack@devstackvm:~$ cat /proc/3070/limits  | grep "Max open files"
> Max open files1024 4096
> files
> stack@devstackvm:~$
> stack@devstackvm:~$ echo -n "Max open files=8192:unlimited"  >
> /proc/3070/limits
> -bash: /proc/3070/limits: Permission denied
> stack@devstackvm:~$ sudo echo -n "Max open files=8192:unlimited"  >
> /proc/3070/limits
> -bash: /proc/3070/limits: Permission denied
> stack@devstackvm:~$
>
> root@devstackvm:/home/stack# echo -n "Max open files=8192:unlimited" >
> /proc/3070/limits
> bash: echo: write error: Invalid argument
> root@devstackvm:/home/stack#
>
> On Mon, Dec 19, 2016 at 11:27 PM, John Petrini <jpetr...@coredial.com>
> wrote:
>
>> Hi Preshant,
>>
>> You can change the open file limit of the running process by echoing the
>> value to it. For example...
>>
>> echo -n "Max open files=8192:unlimited"  > /proc//limits
>>
>>
>> ___
>>
>> John Petrini
>>
>>
>>
>> On Mon, Dec 19, 2016 at 12:21 PM, Prashant Shetty <
>

Re: [Openstack] Max open files limit for nova-api

[John Petrini]

Hi Preshant,

You can change the open file limit of the running process by echoing the
value to it. For example...

echo -n "Max open files=8192:unlimited"  > /proc//limits


___

John Petrini



On Mon, Dec 19, 2016 at 12:21 PM, Prashant Shetty <
prashantshetty1...@gmail.com> wrote:

> Hi Arne,
> Thanks for your reply. Currently all these services are running on ubuntu
> controller under screen.
> Do we have any option to set the file limit option for n-api service in
> this case?. I am not using systemd in my setup to run these services.
>
> Thanks,
> Prashant
>
> On Mon, Dec 19, 2016 at 10:19 PM, Arne Wiebalck <arne.wieba...@cern.ch>
> wrote:
>
>> Prashant,
>>
>> If this is for systemd, how about changing the nova-api unit file?
>>
>> Something like
>>
>> —>
>> [Service]
>> ...
>> LimitNOFILE=65536
>> <—
>>
>> should do it.
>>
>> Cheers,
>>  Arne
>>
>>
>>
>> On 19 Dec 2016, at 17:23, Prashant Shetty <prashantshetty1...@gmail.com>
>> wrote:
>>
>> Team,
>>
>> I have scale setup and metadata requests are seems to fail from instance.
>> Main reason for failure is "Max open files" limit(1024) set on nova-api
>> service.
>> Though on controller we have set max open file limit of 65k(limit.conf),
>> nova-api always comes up with 1024 limit causing failure.
>>
>> Could someone let me know how can we change the max open files limit of
>> nova-api service?
>>
>> Setup Details:
>>
>> · Single controller
>> · 500 KVM computes
>> · Devstack branch: stable/newton
>> · We have native metadata and dhcp running on platform
>> · 3750 instances
>>
>>
>> stack@controller:/opt/stack/logs$ ps aux | grep nova-api
>> stack 14998 2.2 0.3 272104 121648 pts/8 S+ 09:53 0:14 /usr/bin/python
>> /usr/local/bin/nova-api
>> stack@controller:/opt/stack/logs$
>> stack@controller:/opt/stack/logs$
>> stack@controller:/opt/stack/logs$ cat /proc/14998/limits
>> Limit Soft Limit Hard Limit Units
>> Max cpu time unlimited unlimited seconds
>> Max file size unlimited unlimited bytes
>> Max data size unlimited unlimited bytes
>> Max stack size 8388608 unlimited bytes
>> Max core file size unlimited unlimited bytes
>> Max resident set unlimited unlimited bytes
>> Max processes 128611 128611 processes
>> Max open files 1024 4096 files
>> Max locked memory 65536 65536 bytes
>> Max address space unlimited unlimited bytes
>> Max file locks unlimited unlimited locks
>> Max pending signals 128611 128611 signals
>> Max msgqueue size 819200 819200 bytes
>> Max nice priority 0 0
>> Max realtime priority 0 0
>> Max realtime timeout unlimited unlimited us
>> stack@controller:/opt/stack/logs$
>>
>> n-api:
>>
>> 2016-11-08 18:44:26.168 30069 INFO nova.metadata.wsgi.server
>> [req-fb4d729b-a1cd-4df1-aaf8-3f854a739cce - -] (30069) wsgi exited,
>> is_accepting=True
>> Traceback (most recent call last):
>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py",
>> line 457, in fire_timers
>> timer()
>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py",
>> line 58, in __call__
>> cb(*args, **kw)
>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line
>> 168, in _do_send
>> waiter.switch(result)
>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
>> line 214, in main
>> result = function(*args, **kwargs)
>>   File "/opt/stack/nova/nova/utils.py", line 1066, in context_wrapper
>> return func(*args, **kwargs)
>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line
>> 865, in server
>> client_socket = sock.accept()
>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py",
>> line 214, in accept
>> res = socket_accept(fd)
>>   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py",
>> line 56, in socket_accept
>> return descriptor.accept()
>>   File "/usr/lib/python2.7/socket.py", line 206, in accept
>> sock, addr = self._sock.accept()
>> error: [Errno 24] Too many open files
>>
>> Thanks,
>> Prashant
>>
>> ___
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to : openstack@lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>> --
>> Arne Wiebalck
>> CERN IT
>>
>>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] [fuel] Fuel license details

[John Petrini]

You can view the license on github
https://github.com/openstack/fuel-main/blob/master/LICENSE

___

John Petrini

NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
//   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
*P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

[image: Exceptional people. Proven Processes. Innovative Technology.
Discover CoreDial - watch our video]
<http://cta-redirect.hubspot.com/cta/redirect/210539/4c492538-6e4b-445e-9480-bef676787085>

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Wed, Dec 14, 2016 at 8:10 PM, Luca Cervigni <luca.cervi...@pawsey.org.au>
wrote:

> Hello everyone,
>
> Sorry to post here but there are no information both in the openstack
> documentation nor in the project website on the Fuel license. (pages are
> missing / not still written)
>
> I am reading that fuel is an opensource project maintained by the
> openstack community, but is its license the same Apache 2.0 license ?
>
> Thanks for confirming that
>
> Cheers, Luca
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
> k
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Malformed request URL doesn't match Context's project_id

[John Petrini]

It looks like when you request the token you are not providing tenantName
so you're not generating a token for authenticating to tenant/project you
want to authenticate to. i.e. e5ec75ca491049b2b3d6758269aad07c

___

John Petrini

On Sat, Dec 10, 2016 at 7:43 AM, Jun Hu <jhu_...@outlook.com> wrote:

> Hi Guys:
>
> My openstack version is Mitaka.  I want to use curl to call openstack API
> , but I met a issue. Can you help me ?
>
> [root@ospp9-ctrl1 site-packages(keystone_demo)]# source ~/keystonerc_admin
>
> [root@ospp9-ctrl1 site-packages(keystone_admin)]# curl -s -X POST
> $OS_AUTH_URL/tokens   -H "Content-Type: application/json" -d '{"auth":
> {"tenantName": "", "passwordCredentials": {"username": "'"$OS_USERNAME"'",
> "password": "'"$OS_PASSWORD"'"}}}'  | python -m json.tool
> {
> "access": {
> "metadata": {
> "is_admin": 0,
> "roles": []
> },
> "serviceCatalog": [],
> "token": {
> "audit_ids": [
> "RJrDyNfIShyKTmXu5IWZUw"
> ],
> "expires": "2016-12-10T13:12:15Z",
> "id": "80b3d146c5614c149ccf983b21d93055",
> "issued_at": "2016-12-10T12:12:15.171884Z"
> },
> "user": {
> "id": "ae11a07ef07e47bba4a91d1c3516ac01",
> "name": "admin",
> "roles": [],
> "roles_links": [],
> "username": "admin"
> }
> }
> }
> [root@ospp9-ctrl1 site-packages(keystone_admin)]# export OS_TOKEN=
> 80b3d146c5614c149ccf983b21d93055
> [root@ospp9-ctrl1 site-packages(keystone_admin)]# export OS_PROJECT_ID=
> e5ec75ca491049b2b3d6758269aad07c
> [root@ospp9-ctrl1 site-packages(keystone_admin)]# curl -s -H
> "X-Auth-Token:$OS_TOKEN" http://192.168.122.64:8774/v2/
> e5ec75ca491049b2b3d6758269aad07c/flavors  | python -m json.tool
> {
> "badRequest": {
> "code": 400,
> "message": "Malformed request URL: URL's project_id '
> e5ec75ca491049b2b3d6758269aad07c' doesn't match Context's project_id
> 'None'"
> }
> }
>
> [root@ospp9-ctrl1 site-packages(keystone_admin)]# openstack catalog show
> nova
> +---+---
> +
> | Field | Value
> |
> +---+---
> +
> | endpoints | RegionOne
> |
> |   |   publicURL: http://192.168.122.64:8774/v2/
> e5ec75ca491049b2b3d6758269aad07c   |
> |   |   internalURL: http://192.168.122.64:8774/v2/
> e5ec75ca491049b2b3d6758269aad07c |
> |   |   adminURL: http://192.168.122.64:8774/v2/
> e5ec75ca491049b2b3d6758269aad07c|
> |   |
> |
> | name  | nova
> |
> | type  | compute
> |
> +---+---
> +
>
> 1. After I deep into the code, and have a question : How to get/set the 
> context.project_id?
>
>
>
> [root@ospp9-ctrl1 site-packages(keystone_admin)]# vim
> nova/api/openstack/wsgi.py +721
> ```python
>project_id = action_args.pop("project_id", None)
> context = request.environ.get('nova.context')
> if (context and project_id and (project_id != context.project_id)):
> msg = _("Malformed request URL: URL's project_id
> '%(project_id)s'"
> " doesn't match Context's project_id"
> " '%(context_project_id)s'") % \
> {'project_id': project_id,
>  'context_project_id': context.project_id}
> return Fault(webob.exc.HTTPBadRequest(explanation=msg))
> ```
>
> 2. Why  got empty SeviceCatalog"serviceCatalog": []" but  it is not
> in official docs
> http://developer.openstack.org/api-guide/quick-start/api-
> quick-start.html#openstack-api-quick-guide
>
> --
> 
> Margin Hu
> Love Open Source Software.
> Mobile: (86) 186-8035-6499
> email : jhu_...@outlook.com
> github: http://github.com/todaygood
>  ---
>
>
> ___
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to : openstack@lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] UnsupportedVersion Endpoint does not support RPC version

[John Petrini]

I don't have any further suggestions. Since the 7th parameter was added in
API version 4.2 the issue you're having doesn't really surprise me. However
it seems like you have a workaround for migrating either direction. Is that
not enough for what you're trying to accomplish?

___

John Petrini

NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
//   [image:
Twitter] <https://twitter.com/coredial>   [image: LinkedIn]
<http://www.linkedin.com/company/99631>   [image: Google Plus]
<https://plus.google.com/104062177220750809525/posts>   [image: Blog]
<http://success.coredial.com/blog>
Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
*P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com

[image: Exceptional people. Proven Processes. Innovative Technology.
Discover CoreDial - watch our video]
<http://cta-redirect.hubspot.com/cta/redirect/210539/4c492538-6e4b-445e-9480-bef676787085>

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Wed, Oct 26, 2016 at 9:12 PM, Oisin O'Malley <oisin.omal...@iocane.com.au
> wrote:

> Thanks John,
>
> > I think it may be your API version. See here:
> https://github.com/openstack/nova/blob/2014.2/nova/compute/rpcapi.py#L270
>
> There seems to be a discrepancy in the rpcapi.py "kilo" alias between the
> Kilo and Liberty releases. In the Kilo release the compute alias is
> "kilo=3.40" but in Liberty it is "kilo=4.0". Setting the compute API
> version to 4.0 on the Kilo compute node or removing it entirely appears to
> resolve the issue. Though has either caused or led to another issue.
>
> I've testing and can live migrate from Kilo -> Liberty compute nodes, but
> fails on migrating from  Liberty -> Kilo. I suspect another API version
> issue. With compute API level pined to 4.0 across all Nova nodes,  the
> following error is raised in the Liberty Nova compute nodes log when I try
> live migrate from Liberty to Kilo;
>
>File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 142, in _dispatch_and_reply
>  executor_callback))
>File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 186, in _dispatch
>  executor_callback)
>File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 129, in _do_dispatch
>  result = func(ctxt, **new_args)
>File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in
> wrapped
>  payload)
>File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> 195, in __exit__
>  six.reraise(self.type_, self.value, self.tb)
>File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in
> wrapped
>  return f(self, context, *args, **kw)
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 400, in decorated_function
>  return function(self, context, *args, **kwargs)
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 378, in decorated_function
>  kwargs['instance'], e, sys.exc_info())
>File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> 195, in __exit__
>  six.reraise(self.type_, self.value, self.tb)
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 366, in decorated_function
>  return function(self, context, *args, **kwargs)
>  TypeError: live_migration() takes exactly 7 arguments (6 given)
>
> There appears to of been a few bugs raised previously for this. The 4.2
> API version of live_migration with 7 parameters appears to be getting
> called with the 6 parameters from API version 4.0. I'm not sure why.
>
> https://bugs.launchpad.net/nova/+bug/1595864
> I've unsuccessful tried the suggestion in the above and restarted all
> nova-conductor services to no effect.
>
> Any suggestions?
>
>
> Oisin O'Malley
> Systems Engineer
> Iocane Pty Ltd
> 763 South Road
> Black Forest SA 5035
>
> Office:+61 (8) 8413 1010
> Fax:+61 (8) 8231 2050
> Email:oisin.omal...@iocane.com.au
> Web:www.iocane.com.au
>
> Better for business
>
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] UnsupportedVersion Endpoint does not support RPC version

[John Petrini]

I think it may be your API version. See here:
https://github.com/openstack/nova/blob/2014.2/nova/compute/rpcapi.py#L270



On Wed, Oct 26, 2016 at 1:08 AM, Oisin O'Malley  wrote:

> Hi All,
>
> In the process of a Kilo to Liberty upgrade and cannot live migrate VMs
> between Kilo and Liberty compute hosts. The Control nodes and a single
> Compute node have been upgraded to Liberty. We get the following error in
> the compute nodes log when we attempt to live migrate a instance from the
> Kilo node to the Liberty node;
>
> Traceback (most recent call last):
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 5314, in live_migration
>  block_migration, disk, dest, migrate_data)
>File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line
> 623, in pre_live_migration
>  disk=disk, migrate_data=migrate_data)
>File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
> line 156, in call
> retry=self.retry)
>File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py",
> line 90, in _send
> timeout=timeout, retry=retry)
>File 
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 350, in send
>  retry=retry)
>File 
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 341, in _send
>  raise result
>  RemoteError: Remote error: UnsupportedVersion Endpoint does not support
> RPC version 3.19. Attempted met
> hod: pre_live_migration
>
>
> Traceback (most recent call last):
>File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 142, in _dispatch_and_reply
>  executor_callback))
>File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 186, in _dispatch
>  executor_callback)
>File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
> line 130, in _do_dispatch
>  result = func(ctxt, **new_args)
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 6810, in live_migration
>  migrate_data=migrate_data)
>File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in
> wrapped
>  payload)
>File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> 85, in __exit__
>  six.reraise(self.type_, self.value, self.tb)
>File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in
> wrapped
>  return f(self, context, *args, **kw)
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 361, in decorated_function
>  kwargs['instance'], e, sys.exc_info())
>File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> 85, in __exit__
>  six.reraise(self.type_, self.value, self.tb)
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 349, in decorated_function
>  return function(self, context, *args, **kwargs)
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 5322, in live_migration
>  block_migration, migrate_data)
>File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in
> wrapped
>  payload)
>File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> 85, in __exit__
>  six.reraise(self.type_, self.value, self.tb)
>File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in
> wrapped
>  return f(self, context, *args, **kw)
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 361, in decorated_function
>  kwargs['instance'], e, sys.exc_info())
>File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line
> 85, in __exit__
>  six.reraise(self.type_, self.value, self.tb)
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 349, in decorated_function
>  return function(self, context, *args, **kwargs)
>File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
> 5564, in _rollback_live_migration
>  context, instance, bdm.volume_id, dest)
>File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line
> 715, in remove_volume_connection
>  instance=instance, volume_id=volume_id)
>File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py",
> line 156, in call
>  retry=self.retry)
>File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py",
> line 90, in _send
>  timeout=timeout, retry=retry)
>File 
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 350, in send
>  retry=retry)
>File 
> "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 341, in _send
>  raise result
>  RemoteError: Remote error: UnsupportedVersion Endpoint does not support
> RPC version 3.30. Attempted method: remove_volume_connection
>
>
> Upgrade level has been set to "kilo" on all Nova Control and Compute nodes
> in /etc/nova/nova.conf;
>