Re: [Openstack] Neutron Agent Heartbeat
That's exactly what I ended up doing (raised it to 60s), and it reduced the database load quite a bit. Thanks On Mon, Oct 14, 2013 at 4:48 AM, Mark McClain wrote: > > On Oct 13, 2013, at 1:53 PM, Mina Nagy Zaki wrote: > >> Hello list, >> Neutron agents send a *lot* of heartbeat messages, and each of these >> is written back to the central mysql database. Does neutron support >> writing to memcached instead? (like nova does with service heartbeats) >> > > Neutron does not currently support memcached. I expect improving the agent > framework is something we'll look at for Icehouse. > > >> Also, any other advice on reducing MySQL load? I'm currently seeing >> about 16 queries/sec > > > One way to reduce the number of queries is to increase the "report_interval" > in the agent configuration. Changing this value to a large value does have > some consequences, so you'll need to experiment to determine the best value > for your deployment. > > mark -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] Neutron Agent Heartbeat
Hello list, Neutron agents send a *lot* of heartbeat messages, and each of these is written back to the central mysql database. Does neutron support writing to memcached instead? (like nova does with service heartbeats) Also, any other advice on reducing MySQL load? I'm currently seeing about 16 queries/sec Thanks! -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] Compute Node Resources
Hello list, How does nova decide which compute node it's going to deploy a new VM to? Can I specify that I want to deploy a VM to a specific host? Thanks! -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] Compromised Cloud
Hello list, One of my VMs was compromised and used as a DDoS node causing me problems with my provider. Any advice on how I would go about monitoring such behaviour on the openstack compute node level and/or disabling compromised machines? -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Force network reset
It does not, in my experience. Please note I actually delete the OVS database entirely and recreate it. On Fri, Oct 4, 2013 at 3:59 PM, Jacob Godin wrote: > Restarting the ovs agents should do it > > Sent from my mobile device > > On Oct 4, 2013 10:50 AM, "Mina Nagy Zaki" wrote: >> >> No that does not work. >> The main problem is OVS somehow learns some bad routes. I can't figure >> out the problem yet so as a temporary fix I reset it (delete the >> database and recreate bridges and restart networking). At that point I >> need quantum to re-add all the ports to the proper bridges. >> >> >> On Fri, Oct 4, 2013 at 2:34 PM, Razique Mahroua >> wrote: >> > Not sure but restarting the services might do it…; >> > >> > Le 4 oct. 2013 à 14:08, Mina Nagy Zaki a écrit : >> > >> >> How do I force quantum to recreate the network ports for a machine? >> >> That is recreate the Openvswitch configuration for a machine? >> >> >> >> -- >> >> Mina Nagy Zaki >> >> >> >> ___ >> >> Mailing list: >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >> Post to : openstack@lists.openstack.org >> >> Unsubscribe : >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> > >> >> >> >> -- >> Mina Nagy Zaki >> >> ___ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Force network reset
No that does not work. The main problem is OVS somehow learns some bad routes. I can't figure out the problem yet so as a temporary fix I reset it (delete the database and recreate bridges and restart networking). At that point I need quantum to re-add all the ports to the proper bridges. On Fri, Oct 4, 2013 at 2:34 PM, Razique Mahroua wrote: > Not sure but restarting the services might do it…; > > Le 4 oct. 2013 à 14:08, Mina Nagy Zaki a écrit : > >> How do I force quantum to recreate the network ports for a machine? >> That is recreate the Openvswitch configuration for a machine? >> >> -- >> Mina Nagy Zaki >> >> ___ >> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] Force network reset
How do I force quantum to recreate the network ports for a machine? That is recreate the Openvswitch configuration for a machine? -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Intermittent connectivity problems from qrouter namespace
That's a great guess :D I did manage to figure this out earlier and it took me a day of understanding OVS and following datapath/flow dumps until I realized that the non-standard configuration I had was causing the creation of a GRE tunnel from the machine to itself (on another IP). Thanks! On Mon, Aug 26, 2013 at 5:24 PM, Darragh O'Reilly wrote: > > > badly configured tunnels can cause the symptoms you see. So you > could focus on that if your quantum network is using gre tunnels. > > Run 'ovs-vsctl show' on each node and check the remote_ip of each > port on br-tun. > > >>____ >> From: Mina Nagy Zaki >>To: openstack >>Sent: Saturday, 24 August 2013, 11:38 >>Subject: [Openstack] Intermittent connectivity problems from qrouter >>namespace >> >> >>I'm using nova 2013.1.2 with quantum/neutron. >> >>I have a router with a fixed net and an external gateway. There's >>unexplained packet loss from the router's network namespace to the >>VM's fixed IPs. I've seen it with ARP requests, ping requests, and >>pretty much everything else. Sometimes they just don't go through for >>a while. I've also seen a pattern of 2s, 1s, 0.5s ping times on the >>first 3 packets, then normal ping times of <1ms. >>VM -> external network traffic is working great though. >> >>I've tried monitoring the OVS flows and they don't seem to change. >>I've been tcpdumping the interfaces, and packets simply fail to make >>it over the qbrXXXXX bridge sometimes. >> >>Also, after a 'warm up' period of trying to ping the machines and >>failing, once a few packets make it through it starts working >>semi-reliably. >> >>I'm lost. Where do I start with this >> >>-- >>Mina Nagy Zaki >> >>___ >>Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>Post to : openstack@lists.openstack.org >>Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >> >> -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
[Openstack] Intermittent connectivity problems from qrouter namespace
I'm using nova 2013.1.2 with quantum/neutron. I have a router with a fixed net and an external gateway. There's unexplained packet loss from the router's network namespace to the VM's fixed IPs. I've seen it with ARP requests, ping requests, and pretty much everything else. Sometimes they just don't go through for a while. I've also seen a pattern of 2s, 1s, 0.5s ping times on the first 3 packets, then normal ping times of <1ms. VM -> external network traffic is working great though. I've tried monitoring the OVS flows and they don't seem to change. I've been tcpdumping the interfaces, and packets simply fail to make it over the qbrX bridge sometimes. Also, after a 'warm up' period of trying to ping the machines and failing, once a few packets make it through it starts working semi-reliably. I'm lost. Where do I start with this -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] Why would ovs_ofctl and ovs_vsctl disagree about br-ex being a bridge?
ex): addr:9a:08:6e:15:1b:58 > config: 0 > state: 0 > current:10GB-FD COPPER > 13(patch-tun): addr:be:fc:aa:4c:31:70 > config: 0 > state: 0 > LOCAL(br-int): addr:b6:f1:d0:a7:8d:49 > config: PORT_DOWN > state: LINK_DOWN > OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0 > > > Does anyone have any idea why ovs-vsctl and ovs-ofctl would disagree > about the bridginess br-ex ? > Also, am I on the right track trying to configure a flat network, > whilst using gre for tenant (internal) networks? > > Right now we only have 10 ip addresses available for this project, > all drawn from the one publicly routable /24. > > > Many thanks for reading all of this! > Andrew. > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] DHCP Replies
I have a single node setup so far. I manually removed all the tag ovs
properties on the ports and it worked. I'm not using VLANs, so I'm not
sure why the quantum OVS plugin is adding tags.
tcpdump of br-int and dhcp tap:
I started ping from inside the dhcp namespace. I get destination
unreachable for a while, then after a few seconds the packets start
making it through.
tcpdump br-int shows me arp requests for 10.0.0.1 from 10.0.0.5 (dhcp tap)
the arp replies do not show up, and the ping icmp packets do not show up
tcpdump of dhcp tap captures all packets as expected (arp and icmp
echo requests and replies)
OVS plugin config:
[OVS]
integration_bridge = br-int
tenant_network_type = gre
enable_tunneling = True
network_vlan_ranges =
tunnel_bridge = br-tun
tunnel_id_ranges = 1:1000
local_ip = xxx.xxx.xxx.xxx
root@box1 ~ # ovs-vsctl show
6846b700-476a-4995-b770-c9d37f11f92d
Bridge br-tun
Port br-tun
Interface br-tun
type: internal
Port "gre-1"
Interface "gre-1"
type: gre
options: {in_key=flow, out_key=flow,
remote_ip="111.111.111.111"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Bridge br-int
Port "qvof0b52659-8c"
tag: 4095
Interface "qvof0b52659-8c"
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "qvo9a42732d-56"
tag: 1
Interface "qvo9a42732d-56"
Port "qr-7f13795a-6c"
tag: 1
Interface "qr-7f13795a-6c"
type: internal
Port "tap32f910cf-c8"
tag: 4095
Interface "tap32f910cf-c8"
type: internal
Port int-br-ex
Interface int-br-ex
Port br-int
Interface br-int
type: internal
Port "tapef814634-b1"
tag: 1
Interface "tapef814634-b1"
type: internal
Bridge br-ex
Port "qg-22f43415-49"
Interface "qg-22f43415-49"
type: internal
Port br-ex
Interface br-ex
type: internal
Port "eth0"
Interface "eth0"
ovs_version: "1.10.1"
root@box1 ~ # ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:064d98b2b243
n_tables:254, n_buffers:256
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: OUTPUT SET_VLAN_VID SET_VLAN_PCP STRIP_VLAN SET_DL_SRC
SET_DL_DST SET_NW_SRC SET_NW_DST SET_NW_TOS SET_TP_SRC SET_TP_DST
ENQUEUE
1(qvo9a42732d-56): addr:3e:71:71:5c:08:fe
config: 0
state: 0
current:10GB-FD COPPER
speed: 1 Mbps now, 0 Mbps max
2(tap32f910cf-c8): addr:5e:70:61:0d:4b:f8
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
4(patch-tun): addr:22:f5:35:76:2a:fc
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
10(tapef814634-b1): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
14(qr-7f13795a-6c): addr:00:00:00:00:00:00
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
LOCAL(br-int): addr:06:4d:98:b2:b2:43
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0
root@box1 ~ # ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=2829.897s, table=0, n_packets=0, n_bytes=0,
idle_age=2829, priority=2,in_port=2 actions=drop
cookie=0x0, duration=2896.855s, table=0, n_packets=577,
n_bytes=51006, idle_age=286, priority=1 actions=NORMAL
On Wed, Aug 21, 2013 at 10:56 AM, Sushma Korati
wrote:
> Hi Mina Nagy,
>
> We faced smilar issue with VLAN + Openvswitch.
> Are you using VLAN?
> Can you please paste your OVS-configuration and output of below commands in
> compute and quantum node.
> tcpdump and ovs-ofctl dump-flows on dhcp tap, int-br-int, br-int
>
> Regards,
> Sushma Korati
> sushma_kor...@persistent.co.in
> Persistent Systems Ltd. | Partners in Innovation | www.persistentsys.com
> P Please consider your environmental responsibility: Before printing this
> e-mail or any other document, ask yourself whether you need a hard copy.
>
>
>
>
> From: Mina Nagy Zaki [mnz...@gmail.com]
> Sent: Wednesday, August 21, 2013 1:37 PM
> To: openstack@lists.openstack.org
> Subject: [Openstack] DHCP Replies
>
> Hello,
> I have working network configuration, VMs have access to the external
> network, hosts have ac
[Openstack] DHCP Replies
Hello, I have working network configuration, VMs have access to the external network, hosts have access to VMs. But DHCP replies are not making it back into the VMs. tcpdump and iptables tracing show me that the requests make it through just fine, but the replies don't make it out of the qdhcp- namespace (the go out the tap interface there but I'm not sure what happens to them next) How should I go about debugging this? Thanks! -- Mina Nagy Zaki ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
