Re: [openstack-dev] [neutron][L3][HA] 2 masters after reboot of node
Thanks Kevin. I added it to get_router_ids [1], which is called when full_sync flag is set(i.e when agent is AGENT_REVIVED, updated or started) and not in get_routers/sync_routers. [1] https://review.openstack.org/#/c/470905/ On Sat, May 27, 2017 at 2:54 AM, Kevin Benton wrote: > I recommend a completely new RPC endpoint to trigger this behavior that > the L3 agent calls before sync routers. Don't try to add it to sync routers > which is already quite complex. :) > > On Fri, May 26, 2017 at 7:53 AM, Anil Venkata > wrote: > >> Thanks Kevin, Agree with you. I will try to implement this suggestion. >> >> On Fri, May 26, 2017 at 7:01 PM, Kevin Benton wrote: >> >>> Just triggering a status change should just be handled as a port update >>> on the agent side which shouldn't interrupt any existing flows. So an l3 >>> agent reboot should be safe in this case. >>> >>> On May 26, 2017 6:06 AM, "Anil Venkata" wrote: >>> On Fri, May 26, 2017 at 6:14 PM, Kevin Benton wrote: > Perhaps when the L3 agent starts up we can have it explicitly set the > port status to DOWN for all of the HA ports on that node. Then we are > guaranteed that when they go to ACTIVE it will be because the L2 agent has > wired the ports. > > Thanks Kevin. Will it create dependency of dataplane on controlplane. For example, if the node is properly configured(l2 agent wired up, keepalived configured, VRRP exchange happening) but user just restarted only l3 agent, then with the suggestion we won't break l2 connectivity(leading to multiple HA masters) by re configuring again? Or is there a way server can detect that node(not only agent) is down and set port status? > On Fri, May 26, 2017 at 5:27 AM, Anil Venkata > wrote: > >> This is regarding https://bugs.launchpad.net/neutron/+bug/1597461 >> Earlier to fix this, we added code [1] to spawn keepalived only when >> HA network port status is active. >> >> But, on reboot, node will get HA network port's status as ACTIVE from >> server(please see comment [2]), >> though l2 agent might not have wired[3] the port, resulting in >> spawning keepalived. Any suggestions >> how l3 agent can detect that l2 agent has not wired the port and >> then avoid spawning keepalived? >> >> [1] https://review.openstack.org/#/c/357458/ >> [2] https://bugs.launchpad.net/neutron/+bug/1597461/comments/26 >> [3] l2 agent wiring means setting up ovs flows on br-tun to make port >> usable >> >> Thanks >> Anilvenkata >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.op >> enstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.op > enstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.op enstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.op >>> enstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][L3][HA] 2 masters after reboot of node
I recommend a completely new RPC endpoint to trigger this behavior that the L3 agent calls before sync routers. Don't try to add it to sync routers which is already quite complex. :) On Fri, May 26, 2017 at 7:53 AM, Anil Venkata wrote: > Thanks Kevin, Agree with you. I will try to implement this suggestion. > > On Fri, May 26, 2017 at 7:01 PM, Kevin Benton wrote: > >> Just triggering a status change should just be handled as a port update >> on the agent side which shouldn't interrupt any existing flows. So an l3 >> agent reboot should be safe in this case. >> >> On May 26, 2017 6:06 AM, "Anil Venkata" wrote: >> >>> On Fri, May 26, 2017 at 6:14 PM, Kevin Benton wrote: >>> Perhaps when the L3 agent starts up we can have it explicitly set the port status to DOWN for all of the HA ports on that node. Then we are guaranteed that when they go to ACTIVE it will be because the L2 agent has wired the ports. >>> Thanks Kevin. Will it create dependency of dataplane on controlplane. >>> For example, if the node is properly configured(l2 agent wired up, >>> keepalived configured, VRRP exchange happening) but user just restarted >>> only l3 agent, then with the suggestion we won't break l2 >>> connectivity(leading to multiple HA masters) by re configuring again? >>> >>> Or is there a way server can detect that node(not only agent) is down >>> and set port status? >>> >>> On Fri, May 26, 2017 at 5:27 AM, Anil Venkata wrote: > This is regarding https://bugs.launchpad.net/neutron/+bug/1597461 > Earlier to fix this, we added code [1] to spawn keepalived only when > HA network port status is active. > > But, on reboot, node will get HA network port's status as ACTIVE from > server(please see comment [2]), > though l2 agent might not have wired[3] the port, resulting in > spawning keepalived. Any suggestions > how l3 agent can detect that l2 agent has not wired the port and > then avoid spawning keepalived? > > [1] https://review.openstack.org/#/c/357458/ > [2] https://bugs.launchpad.net/neutron/+bug/1597461/comments/26 > [3] l2 agent wiring means setting up ovs flows on br-tun to make port > usable > > Thanks > Anilvenkata > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.op > enstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.op enstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.op >>> enstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][L3][HA] 2 masters after reboot of node
Thanks Kevin, Agree with you. I will try to implement this suggestion. On Fri, May 26, 2017 at 7:01 PM, Kevin Benton wrote: > Just triggering a status change should just be handled as a port update on > the agent side which shouldn't interrupt any existing flows. So an l3 agent > reboot should be safe in this case. > > On May 26, 2017 6:06 AM, "Anil Venkata" wrote: > >> On Fri, May 26, 2017 at 6:14 PM, Kevin Benton wrote: >> >>> Perhaps when the L3 agent starts up we can have it explicitly set the >>> port status to DOWN for all of the HA ports on that node. Then we are >>> guaranteed that when they go to ACTIVE it will be because the L2 agent has >>> wired the ports. >>> >>> >> Thanks Kevin. Will it create dependency of dataplane on controlplane. For >> example, if the node is properly configured(l2 agent wired up, keepalived >> configured, VRRP exchange happening) but user just restarted only l3 agent, >> then with the suggestion we won't break l2 connectivity(leading to multiple >> HA masters) by re configuring again? >> >> Or is there a way server can detect that node(not only agent) is down and >> set port status? >> >> >>> On Fri, May 26, 2017 at 5:27 AM, Anil Venkata >>> wrote: >>> This is regarding https://bugs.launchpad.net/neutron/+bug/1597461 Earlier to fix this, we added code [1] to spawn keepalived only when HA network port status is active. But, on reboot, node will get HA network port's status as ACTIVE from server(please see comment [2]), though l2 agent might not have wired[3] the port, resulting in spawning keepalived. Any suggestions how l3 agent can detect that l2 agent has not wired the port and then avoid spawning keepalived? [1] https://review.openstack.org/#/c/357458/ [2] https://bugs.launchpad.net/neutron/+bug/1597461/comments/26 [3] l2 agent wiring means setting up ovs flows on br-tun to make port usable Thanks Anilvenkata __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.op enstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.op >>> enstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][L3][HA] 2 masters after reboot of node
Just triggering a status change should just be handled as a port update on the agent side which shouldn't interrupt any existing flows. So an l3 agent reboot should be safe in this case. On May 26, 2017 6:06 AM, "Anil Venkata" wrote: > On Fri, May 26, 2017 at 6:14 PM, Kevin Benton wrote: > >> Perhaps when the L3 agent starts up we can have it explicitly set the >> port status to DOWN for all of the HA ports on that node. Then we are >> guaranteed that when they go to ACTIVE it will be because the L2 agent has >> wired the ports. >> >> > Thanks Kevin. Will it create dependency of dataplane on controlplane. For > example, if the node is properly configured(l2 agent wired up, keepalived > configured, VRRP exchange happening) but user just restarted only l3 agent, > then with the suggestion we won't break l2 connectivity(leading to multiple > HA masters) by re configuring again? > > Or is there a way server can detect that node(not only agent) is down and > set port status? > > >> On Fri, May 26, 2017 at 5:27 AM, Anil Venkata >> wrote: >> >>> This is regarding https://bugs.launchpad.net/neutron/+bug/1597461 >>> Earlier to fix this, we added code [1] to spawn keepalived only when HA >>> network port status is active. >>> >>> But, on reboot, node will get HA network port's status as ACTIVE from >>> server(please see comment [2]), >>> though l2 agent might not have wired[3] the port, resulting in spawning >>> keepalived. Any suggestions >>> how l3 agent can detect that l2 agent has not wired the port and >>> then avoid spawning keepalived? >>> >>> [1] https://review.openstack.org/#/c/357458/ >>> [2] https://bugs.launchpad.net/neutron/+bug/1597461/comments/26 >>> [3] l2 agent wiring means setting up ovs flows on br-tun to make port >>> usable >>> >>> Thanks >>> Anilvenkata >>> >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.op >>> enstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][L3][HA] 2 masters after reboot of node
On Fri, May 26, 2017 at 6:14 PM, Kevin Benton wrote: > Perhaps when the L3 agent starts up we can have it explicitly set the port > status to DOWN for all of the HA ports on that node. Then we are guaranteed > that when they go to ACTIVE it will be because the L2 agent has wired the > ports. > > Thanks Kevin. Will it create dependency of dataplane on controlplane. For example, if the node is properly configured(l2 agent wired up, keepalived configured, VRRP exchange happening) but user just restarted only l3 agent, then with the suggestion we won't break l2 connectivity(leading to multiple HA masters) by re configuring again? Or is there a way server can detect that node(not only agent) is down and set port status? > On Fri, May 26, 2017 at 5:27 AM, Anil Venkata > wrote: > >> This is regarding https://bugs.launchpad.net/neutron/+bug/1597461 >> Earlier to fix this, we added code [1] to spawn keepalived only when HA >> network port status is active. >> >> But, on reboot, node will get HA network port's status as ACTIVE from >> server(please see comment [2]), >> though l2 agent might not have wired[3] the port, resulting in spawning >> keepalived. Any suggestions >> how l3 agent can detect that l2 agent has not wired the port and >> then avoid spawning keepalived? >> >> [1] https://review.openstack.org/#/c/357458/ >> [2] https://bugs.launchpad.net/neutron/+bug/1597461/comments/26 >> [3] l2 agent wiring means setting up ovs flows on br-tun to make port >> usable >> >> Thanks >> Anilvenkata >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [neutron][L3][HA] 2 masters after reboot of node
Perhaps when the L3 agent starts up we can have it explicitly set the port status to DOWN for all of the HA ports on that node. Then we are guaranteed that when they go to ACTIVE it will be because the L2 agent has wired the ports. On Fri, May 26, 2017 at 5:27 AM, Anil Venkata wrote: > This is regarding https://bugs.launchpad.net/neutron/+bug/1597461 > Earlier to fix this, we added code [1] to spawn keepalived only when HA > network port status is active. > > But, on reboot, node will get HA network port's status as ACTIVE from > server(please see comment [2]), > though l2 agent might not have wired[3] the port, resulting in spawning > keepalived. Any suggestions > how l3 agent can detect that l2 agent has not wired the port and > then avoid spawning keepalived? > > [1] https://review.openstack.org/#/c/357458/ > [2] https://bugs.launchpad.net/neutron/+bug/1597461/comments/26 > [3] l2 agent wiring means setting up ovs flows on br-tun to make port > usable > > Thanks > Anilvenkata > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [neutron][L3][HA] 2 masters after reboot of node
This is regarding https://bugs.launchpad.net/neutron/+bug/1597461 Earlier to fix this, we added code [1] to spawn keepalived only when HA network port status is active. But, on reboot, node will get HA network port's status as ACTIVE from server(please see comment [2]), though l2 agent might not have wired[3] the port, resulting in spawning keepalived. Any suggestions how l3 agent can detect that l2 agent has not wired the port and then avoid spawning keepalived? [1] https://review.openstack.org/#/c/357458/ [2] https://bugs.launchpad.net/neutron/+bug/1597461/comments/26 [3] l2 agent wiring means setting up ovs flows on br-tun to make port usable Thanks Anilvenkata __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
