Some more info added to Jakub's excellent report :)
New kernel Ubuntu-4.4.0-89.112HEADUbuntu-4.4.0-89.112master was tagged 9 days ago (07/31/2017) [0]. >From a quick look, the only commit around this function is [1]. [0] http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=64de31ed97a03ec1b86fd4f76e445506dce55b02 [1] http://kernel.ubuntu.com/git/ubuntu/ubuntu-xenial.git/commit/?id=2ad4caea651e1cc0fc86111ece9f9d74de825b78 On Wed, Aug 9, 2017 at 3:29 PM, Jakub Libosvar <jlibo...@redhat.com> wrote: > Daniel Alvarez and I spent some time looking at it and the culprit was > finally found. > > tl;dr > > We updated a kernel on machines to one containing bug when creating > conntrack entries which makes functional tests stuck. More info at [4]. > > For now, I sent a patch [5] to disable for now jobs that create > conntrack entries manually, it needs update of commit message. Once it > merges, we an enable back functional job to voting to avoid regressions. > > Is it possible to switch used image for jenkins machines to use back the > older version? Any other ideas how to deal with the kernel bug? > > Thanks > Jakub > > [5] https://review.openstack.org/#/c/492068/1 > > On 07/08/2017 11:52, Jakub Libosvar wrote: > > Hi all, > > > > as per grafana [1] the functional job is broken. Looking at logstash [2] > > it started happening consistently since 2017-08-03 16:27. I didn't find > > any particular patch in Neutron that could cause it. > > > > The culprit is that ovsdb starts misbehaving [3] and then we retry calls > > indefinitely. We still use 2.5.2 openvswitch as we had before. I opened > > a bug [4] and started investigation, I'll update my findings there. > > > > I think at this point there is no reason to run "recheck" on your > patches. > > > > Thanks, > > Jakub > > > > [1] > > http://grafana.openstack.org/dashboard/db/neutron-failure- > rate?panelId=7&fullscreen > > [2] http://bit.ly/2vdKMwy > > [3] > > http://logs.openstack.org/14/488914/8/check/gate-neutron- > dsvm-functional-ubuntu-xenial/75d7482/logs/openvswitch/ovsdb-server.txt.gz > > [4] https://bugs.launchpad.net/neutron/+bug/1709032 > > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
