[Openstack-operators] MessagingTimeout in block live-migration due to long image fetch operation

Hi everyone, (running Mitaka) When doing block live-migration, if the image / backing file is not present at destination host, sometimes pre-live migration fails after 60 seconds as shown below. Retrying the migration to the same destination host succeeds. It seems that an rpc_response_timeout

Re: [Openstack-operators] [Openstack] Strange: lost physical connectivity to compute hosts when using native (ryu) openflow interface

Tue, May 30, 2017 at 9:49 AM, Gustavo Randich < > gustavo.rand...@gmail.com> wrote: > >> While dumping OVS flows as you suggested, we finally found the cause of >> the problem: our br-ex OVS bridge lacked the secure fail mode configuration. >> >> May be the is

Re: [Openstack-operators] [Openstack] Strange: lost physical connectivity to compute hosts when using native (ryu) openflow interface

vswitch-agent -> access to VMs is restored Thank you! On Fri, Apr 28, 2017 at 5:07 PM, Kevin Benton <ke...@benton.pub> wrote: > With the network down, does ovs-vsctl show that it is connected to the > controller? > > On Fri, Apr 28, 2017 at 2:21 PM, Gustavo Randich <

Re: [Openstack-operators] [Openstack] Strange: lost physical connectivity to compute hosts when using native (ryu) openflow interface

pid --detach --monitor Thanks On Fri, Apr 28, 2017 at 5:00 AM, Kevin Benton <ke...@benton.pub> wrote: > Are you using an of_listen_address value of an interface being brought > down? > > On Apr 25, 2017 17:34, "Gustavo Randich" <gustavo.rand...@gmail.com> &g

[Openstack-operators] query NUMA topology via API

Hi, is there any way to query via Compute API the NUMA topology of a compute node, and free ram/cpu of each NUMA cell? Thanks! ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org

Re: [Openstack-operators] [Openstack] DVR ARP cache update loop delaying launch of metadata proxy

Thanks Brian for the link, it's exactly what I was looking for. On Thu, Dec 15, 2016 at 5:29 PM, Brian Haley <brian.ha...@hpe.com> wrote: > On 12/13/2016 02:45 PM, Gustavo Randich wrote: > >> Hi Openstackers, >> >> We have the folowing issue (using Mitaka /

[Openstack-operators] DVR ARP cache update loop delaying launch of metadata proxy

Hi Openstackers, We have the folowing issue (using Mitaka / DVR / Xenial), perhaps someone can help ;) When our hosts boots up, the ARP cache population loop of L3 Agent is delaying the start of neutron-ns-metadata-proxy for around a minute -- see logs below; then, when nova-compute launches

[Openstack-operators] sync power states and externally shut off VMs

When a VM is shutdown without using nova API (kvm process down, libvirt failed to start instance on host boot, etc.), Openstack "freezes" the shutdown power state in the DB, and then re-applies it if the VM is not started via API, e.g.: # virsh shutdown [ sync power states -> stop instance via

Re: [Openstack-operators] problems with iscsi multipath (Ubuntu 16 / Mitaka)

ch file or directory Maybe this is related to missing wwid in /etc/multipath/wwids? We are running kernel 4.6 Any hint will be appreciated Gustavo On Friday, 12 August 2016, Gustavo Randich <gustavo.rand...@gmail.com> wrote: > Regarding the cleanup in detach, I found that the python-os

Re: [Openstack-operators] problems with iscsi multipath (Ubuntu 16 / Mitaka)

I partially applied those changes and could make it work (clean up) On Fri, Aug 12, 2016 at 5:47 PM, Gustavo Randich <gustavo.rand...@gmail.com> wrote: > Hi, > > Is someone using iscsi multipath in Ubuntu 16 / Mitaka? Things that work > OK for us in Ubuntu 14 / Icehouse

[Openstack-operators] problems with iscsi multipath (Ubuntu 16 / Mitaka)

Hi, Is someone using iscsi multipath in Ubuntu 16 / Mitaka? Things that work OK for us in Ubuntu 14 / Icehouse seem to be broken. Basically, the WWID of the volume is not automatically placed in /etc/multipath/wwids, and if we add it manually, we can attach with multipath, but the detach never

Re: [Openstack-operators] systemd and duplicate logs -- /var/log/syslog and /var/log/nova/nova-compute.log

ng system to not ouput logs prevented > that from happening. > > > > ___ > > Kris Lindgren > > Senior Linux Systems Engineer > > GoDaddy > > > > *From: *Gustavo Randich <gustavo.rand...@gmail.com> > *Date: *Tuesday, August 9, 2016 at 3:52 PM > *To: *"opensta

[Openstack-operators] systemd and duplicate logs -- /var/log/syslog and /var/log/nova/nova-compute.log

Hi guys, We want to be able to forward nova-compute's log to a central rsyslog but at the same time mantain the local "/var/log/nova/nova-compute.log". In Icehouse we achieved this with the following configuration in "/etc/rsyslog.d/60-nova.conf": *.*;local0.none,auth,authpriv.none

Re: [Openstack-operators] migration leaves instance in VERIFY_RESIZE and does not clean up source host resources

Sorry, I forgot to run 'nova resize-confirm' On Mon, Aug 8, 2016 at 6:17 PM, Gustavo Randich <gustavo.rand...@gmail.com> wrote: > Hi folks, > > I'm using Mitaka, and when running "nova migrate ", once > the instance is migrated OK to other host, it remains forev

[Openstack-operators] NTP client in guest not reaching NTP server in host (Neutron/DVR)

Hi, With nova-network, we were running NTP in guests against an NTP server in the host (following this advice). Now tih Neutron/DVR, we are not reaching NTP in the host because it is

Re: [Openstack-operators] Reaching VXLAN tenant networks from outside (without floating IPs)

> Regards, > Mike > > > > From:Gustavo Randich <gustavo.rand...@gmail.com> > To:Mike Spreitzer/Watson/IBM@IBMUS > Cc:"openst...@lists.openstack.org" <openst...@lists.openstack.org>, > "openstack-operators@lists.openstack.

Re: [Openstack-operators] Reaching VXLAN tenant networks from outside (without floating IPs)

t network. > > The normal configuration for routers on the internet is to not forward > traffic to the RFC 1918 addresses. I do not recall how the Neutron routers > handle packets addressed to those addresses from sources on the "outside". > > Regards, > Mike > > &g

Re: [Openstack-operators] [Openstack] Reaching VXLAN tenant networks from outside (without floating IPs)

Thanks Mike, works OK! Simple IP routing :) On Thu, Jun 30, 2016 at 5:17 PM, Gustavo Randich <gustavo.rand...@gmail.com> wrote: > Thank you, I'll test this approach and see... > > > > On Thu, Jun 30, 2016 at 3:04 PM, Rick Jones <rick.jon...@hpe.com> wrote: > &

Re: [Openstack-operators] [Openstack] Reaching VXLAN tenant networks from outside (without floating IPs)

Thank you, I'll test this approach and see... On Thu, Jun 30, 2016 at 3:04 PM, Rick Jones wrote: > On 06/30/2016 10:32 AM, Mike Spreitzer wrote: > >> No, those routers are routers. If one of them gets a packet, the router >> will forward the packet as usual for a router.

Re: [Openstack-operators] Reaching VXLAN tenant networks from outside (without floating IPs)

Mike, as far as I know those routers allow only outgoing traffic, i.e. VM can see external networks, but those external networks cannot connect to VM if it doesn't have a FIP, am I right? Thanks! Gustavo On Wed, Jun 29, 2016 at 7:24 PM, Mike Spreitzer <mspre...@us.ibm.com> wrote: >

[Openstack-operators] DVR (kilo) connecting subnets in different routers

Hi, (Using Kilo/DVR) I want to connect two tenant subnets which are connected to different routers. I had to add a port in tenant-network1 as an interface in router2 and a port in tenant-network2 as an interface in router1 to make it work. I.e., if I only add network1 to router2, DVR does not

[Openstack-operators] problem with DVR in Kilo and floating IPs

Hi, Using DVR in Kilo, I've the following issue: - VM1 is in tenant network 1 (fixed IP 10.97.2.4) - VM2 is in tenant network 2 (fixed IP 10.97.0.4) - a router connects both networks - VM1 and VM2 both have floating IPs - I can ping from VM1 to VM2 using fixed / internal IP - I cannot SSH from

[Openstack-operators] nova-compute-lxd on kilo / trusty

Hi, Is it possible to install nova-compute-lxd with Kilo and Trusty? Thanks! ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Can I authenticate a Kilo service against an Icehouse identity server?

solved; I was hitting this bug: https://bugs.launchpad.net/keystone/+bug/1343579 :) On Mon, May 9, 2016 at 2:24 PM, Gustavo Randich <gustavo.rand...@gmail.com> wrote: > ok here is a little more log detail... > > 2016-05-09 13:18:53.246 8014 INFO eventlet.wsgi.server [-]

Re: [Openstack-operators] Can I authenticate a Kilo service against an Icehouse identity server?

e": "application/vnd.openstack.identity-v3+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v3+xml"}], "id": "v3.0", "links": [{"href": "https://icehouse-api:35357/v3/;,

[Openstack-operators] Can I authenticate a Kilo service against an Icehouse identity server?

Hi, Is it possible to integrate python-keystoneclient as of Kilo release (1.2.0) with keystone Icehouse (2014.1)? Specifically, I'm trying to make cinder-api (Kilo) authenticate against an Icehouse keystone server, but the following appears in the log: 2016-05-09 12:55:06.846 7623 INFO

Re: [Openstack-operators] [Scale][Performance] / compute_nodes ratio experience

We recently had a power outage, and perhaps one of the scenarios of controller capacity planning is starting all of the compute nodes at once or in large batches (when power was restored). We painfully learned about our nova-conductor being low on workers/cores, but still we doubted whether it

Re: [Openstack-operators] nova-conductor scale out

s/is constantly crushing the cpu on those > servers. > > ___ > Kris Lindgren > Senior Linux Systems Engineer > GoDaddy > > From: David Medberry <openst...@medberry.net> > Date: Tuesday, March 15, 201

Re: [Openstack-operators] nova-conductor scale out

PD: 32 cores On Tue, Mar 15, 2016 at 12:37 PM, Gustavo Randich <gustavo.rand...@gmail.com > wrote: > We are melting right now (rpc timeouts, rabbitmq connection timeouts, high > load on controller, etc.): we are running 375 compute nodes, and only one > controller (on vmware)

[Openstack-operators] nova-conductor scale out

Hi, Simple question: can I deploy nova-conductor across several servers? (Icehouse) Because we are reaching a limit in our controller node ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org

[Openstack-operators] VXLAN and jumbo frames: performance gains and configuration options

Hi, Scenario: Kilo / VXLAN / DVR Networking installation documentation mentions that when using tunneling protocols, ideally, we could use jumbo frames on the physical network, but then describes a "conservative" approach which is to decrease the MTU size of the guests' interfaces (i.e. to

[Openstack-operators] instances floating IPs not reachable while restarting nova-network

Hi everybody, (still using nova-network in production... :) Using nova-network (icehouse), multi-host, FlatDHCPManager Is it expectable to experience an interruption of various seconds in instances' floating IP reachability when nova-network is restarted and repopulates iptables' NAT

Re: [Openstack-operators] Neutron/DVR scalability of one giant single tenant VS multiple tenants

Benton On Thu, May 14, 2015 at 9:15 AM, Gustavo Randich gustavo.rand...@gmail.com wrote: Hi! We are evaluating the migration of our private cloud of several thousand VMs from multi-host nova-network to neutron/DVR. For historical reasons, we currently use a single tenant because group

Re: [Openstack-operators] how to filter outgoing VM traffic in icehouse

Hi, sorry, I forgot to mention: I'm using nova-network On Wed, May 13, 2015 at 6:39 PM, Abel Lopez alopg...@gmail.com wrote: Yes, you can define egress security group rules. On May 13, 2015, at 2:32 PM, Gustavo Randich gustavo.rand...@gmail.com wrote: Hi, Is there any way to filter

[Openstack-operators] [DVR] Instances without floating-ip cannot reach external network

Hi everybody, I've just configuerd DVR in Kilo. Everything is fine, except that instances with no floating-ip cannot reach external network. NETWORKS neutron net-create ext-net1 --router:external --provider:physical_network external --provider:network_type flat neutron subnet-create ext-net1

[Openstack-operators] reference configuration values for VXLAN + access to external VLANs

Hi, Is there any reference configuration / install guide for this use case?. I only need the key configuration values for ml2 plugin. - Kilo release - ml2 - DVR + vxlan tunneling - provider network = flat - acces to multiple external VLANs The idea is to use tunneling over a flat

[Openstack-operators] how to attach VMs to external/vlan network directly

Hi, I've tried without success to attach instances directly to external VLAN network using provider:network_type vlan; below are the details. Using provider:network_type flat I made it work. I was basically following this: http://www.s3it.uzh.ch/blog/openstack-neutron-vlan/ Any idea will be

Re: [Openstack-operators] logging to syslog and log file with different levels

based on its level but syslog has an ability to do it. So how about using syslog as nova-compute logging and changing the output by the syslog? We are using the solution to change actions by logging levels. regard, Masa On 2015/02/27 6:31, Gustavo Randich wrote: Hi, Is it possible

[Openstack-operators] logging to syslog and log file with different levels

Hi, Is it possible to configure nova-compute logging with INFO level in nova-compute.log and WARN level to syslog? I couldn't play with options in /etc/nova/logging.conf, because of the folowing problems: - folsom: when I use log_config=/etc/nova/logging.conf, the following error is raised: No

Re: [Openstack-operators] Way to check compute - rabbitmq connectivity

I've been tracking related to this for a while. I haven't really kept up to speed with it, so I don't know the current status. https://bugs.launchpad.net/nova/+bug/856764 From: Kris Lindgren klindg...@godaddy.com Date: Thursday, January 15, 2015 at 12:10 PM To: Gustavo Randich gustavo.rand