[Yahoo-eng-team] [Bug 1978369] Re: [ovn] External Gateway Loop in NB SB DB

2022-06-12 Thread yatin
*** This bug is a duplicate of bug 1973347 ***
https://bugs.launchpad.net/bugs/1973347

This looks duplicate of https://bugs.launchpad.net/neutron/+bug/1973347
and is fixed with
https://review.opendev.org/c/openstack/neutron/+/842147. This should be
backported to stable branches as well.

@Ammad Can you try out the patch and confirm it fixes the issue for
your?

For now i will mark out it as duplicate of other lp i.e 1973347, please
reopen if you still consider it different issue once you check other bug
and the fix.

** This bug has been marked a duplicate of bug 1973347
   OVN revision_number infinite update loop

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1978369

Title:
  [ovn] External Gateway Loop in NB SB DB

Status in neutron:
  New

Bug description:
  Hi,

  I have installed neutron 20.0 and OVN 22.03 on ubuntu 22.04. When I
  create a router and attach external network with it, it generate loop
  thousands of ovn NB and SB DB transection cause the DB size grows.

  In SB

  OVSDB JSON 300 0f200aa6397e53cd203c99e6674bda75bdd53151
  
{"_date":1654929577073,"Multicast_Group":{"9b50bf0f-f9fe-4b9a-9333-fe2d1744575c":{"ports":["uuid","efc3d1a7-56a6-4235-8a29-4d1defdb459c"]}},"_is_diff":true,"_comment":"ovn-northd","Port_Binding":{"efc3d1a7-56a6-4235-8a29-4d1defdb459c":{"external_ids":["map",[["neutron:revision_number","10678"]]]}}}
  OVSDB JSON 402 86de47a7521717bd9ab7182422a6ad9b424c93d0
  
{"_date":1654929577345,"Multicast_Group":{"9b50bf0f-f9fe-4b9a-9333-fe2d1744575c":{"ports":["uuid","efc3d1a7-56a6-4235-8a29-4d1defdb459c"]}},"_is_diff":true,"_comment":"ovn-northd","Port_Binding":{"d34d2dd5-260b-4253-8429-5a7a89f3a500":{"external_ids":["map",[["neutron:revision_number","10679"]]]},"2ce0135e-b9b5-441b-aaae-7ce580bcf600":{"external_ids":["map",[["neutron:revision_number","10679"]]]}}}

  and In NB

  OVSDB JSON 334 e0ee7ff61d595e6151abd694ce2179c11d9e2570
  
{"_date":1654929536919,"_is_diff":true,"Logical_Router_Port":{"a0c2e43e-f4cb-4331-b070-a726b3da7a17":{"external_ids":["map",[["neutron:revision_number","10567"]]]}},"Logical_Switch_Port":{"cc97ca2c-979e-4754-a8d2-4fff0a666df8":{"options":["map",[["mcast_flood_reports","true"],["requested-chassis","kvm01-a1-r17-lhr01.rapid.pk"]]]}}}
  OVSDB JSON 269 dd8f87d8b132415a423b0f020b23f07d2488acba
  
{"_date":1654929536992,"Logical_Switch_Port":{"cc97ca2c-979e-4754-a8d2-4fff0a666df8":{"options":["map",[["mcast_flood_reports","true"],["requested-chassis","kvm01-a1-r17-lhr01.rapid.pk"]]],"external_ids":["map",[["neutron:revision_number","10567"]]]}},"_is_diff":true}
  OVSDB JSON 334 42d2a02531bd91d88b8783a45da47a33b5e3dc94
  
{"_date":1654929537262,"_is_diff":true,"Logical_Router_Port":{"a0c2e43e-f4cb-4331-b070-a726b3da7a17":{"external_ids":["map",[["neutron:revision_number","10568"]]]}},"Logical_Switch_Port":{"cc97ca2c-979e-4754-a8d2-4fff0a666df8":{"options":["map",[["mcast_flood_reports","true"],["requested-chassis","kvm01-a1-r17-lhr01.rapid.pk"]]]}}}
  OVSDB JSON 269 b8454f003de8cb14961aa37d5a557d2490d34049
  
{"_date":1654929537355,"Logical_Switch_Port":{"cc97ca2c-979e-4754-a8d2-4fff0a666df8":{"options":["map",[["mcast_flood_reports","true"],["requested-chassis","kvm01-a1-r17-lhr01.rapid.pk"]]],"external_ids":["map",[["neutron:revision_number","10568"]]]}},"_is_diff":true}
  OVSDB JSON 334 705b3007e83f0646642510903602965a6192fccf
  
{"_date":1654929537648,"_is_diff":true,"Logical_Router_Port":{"a0c2e43e-f4cb-4331-b070-a726b3da7a17":{"external_ids":["map",[["neutron:revision_number","10569"]]]}},"Logical_Switch_Port":{"cc97ca2c-979e-4754-a8d2-4fff0a666df8":{"options":["map",[["mcast_flood_reports","true"],["requested-chassis","kvm01-a1-r17-lhr01.rapid.pk"]]]}}}
  OVSDB JSON 269 4506e6ee9336bf2b8bde3134badbea7d23e72d33

  I also see below logs in ovn-northd.log

  2022-06-11T06:46:55.927Z|00171|northd|WARN|Dropped 650 log messages in last 
60 seconds (most recently, 0 seconds ago) due to excessive rate
  2022-06-11T06:46:55.927Z|00172|northd|WARN|Unknown chassis '' set as 
options:requested-chassis on LSP '426cf7d5-4fd7-4aa9-806b-9dbe170c543e'.
  2022-06-11T06:47:55.941Z|00173|northd|WARN|Dropped 644 log messages in last 
60 seconds (most recently, 0 seconds ago) due to excessive rate
  2022-06-11T06:47:55.941Z|00174|northd|WARN|Unknown chassis '' set as 
options:requested-chassis on LSP '426cf7d5-4fd7-4aa9-806b-9dbe170c543e'.

  
  I have tested it on ubuntu 20.04 via UCA AND 22.04. Below are the test 
scenerio.

  - Two gateway chassis
  - 5 compute nodes

  I have also tested this with one chassis as well, for which I am
  attaching neutron-server.log when I attached external interface to
  router and ovn nb and sb DBs as well.

  I would be happy to provide any further info that is needed.

  Ammad

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1978369/+subscriptions


-- 
Mailing list: https://lau

[Yahoo-eng-team] [Bug 1978444] [NEW] Volume can't be detached if attachment delete api call fails with 504 gateway timeout

2022-06-12 Thread Takashi Kajinami
Public bug reported:

Description
===
When cinder-api is running behind load balancer like haproxy, the load balancer 
can return 504 if it can not receive response from cinder-api within timeout.
When this timeout occurs while detaching a volume, this results in 
un-detachable volume.

 - nova-compute calls delete attachment api in cinder
 - haproxy detects server timeout and returns 504
 - cinder continues processing the API and removes the attachment
 - nova-compute immediately aborts the volume detachment and leaves the bdm
 - when a client tries to detach the volume again, the detachment fails because 
the attachment no longer exists in Nova

See for details https://bugzilla.redhat.com/show_bug.cgi?id=2002643

Steps to reproduce
==
* Stop cinder-volume
* Detach a volume from an instance
* Start cinder-volume
* Detach the volume again

Expected result
===
* Volume can be detached after cinder-volume is recovered

Actual result
===
* Volume can't be detached

Environment
===
* The issue was initially found in stable/train

Logs & Configs
==
* See https://bugzilla.redhat.com/show_bug.cgi?id=2002643#c1

** Affects: nova
 Importance: Undecided
 Assignee: Takashi Kajinami (kajinamit)
 Status: New

** Changed in: nova
 Assignee: (unassigned) => Takashi Kajinami (kajinamit)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1978444

Title:
  Volume can't be detached if attachment delete api call fails with 504
  gateway timeout

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===
  When cinder-api is running behind load balancer like haproxy, the load 
balancer can return 504 if it can not receive response from cinder-api within 
timeout.
  When this timeout occurs while detaching a volume, this results in 
un-detachable volume.

   - nova-compute calls delete attachment api in cinder
   - haproxy detects server timeout and returns 504
   - cinder continues processing the API and removes the attachment
   - nova-compute immediately aborts the volume detachment and leaves the bdm
   - when a client tries to detach the volume again, the detachment fails 
because the attachment no longer exists in Nova

  See for details https://bugzilla.redhat.com/show_bug.cgi?id=2002643

  Steps to reproduce
  ==
  * Stop cinder-volume
  * Detach a volume from an instance
  * Start cinder-volume
  * Detach the volume again

  Expected result
  ===
  * Volume can be detached after cinder-volume is recovered

  Actual result
  ===
  * Volume can't be detached

  Environment
  ===
  * The issue was initially found in stable/train

  Logs & Configs
  ==
  * See https://bugzilla.redhat.com/show_bug.cgi?id=2002643#c1

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1978444/+subscriptions


-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp