[Yahoo-eng-team] [Bug 1937261] Re: python3-msgpack package broken due to outdated cython
This is a problem with python-msgpack in Ubuntu, and does not look like a bug in oslo.privsep or neutron. ** No longer affects: oslo.privsep -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1937261 Title: python3-msgpack package broken due to outdated cython Status in Ubuntu Cloud Archive: Invalid Status in Ubuntu Cloud Archive ussuri series: Fix Released Status in neutron: New Bug description: After a successful upgrade of the control-plance from Train -> Ussuri on Ubuntu Bionic, we upgraded a first compute / network node and immediately ran into issues with Neutron: We noticed that Neutron is extremely slow in setting up and wiring the network ports, so slow it would never finish and throw all sorts of errors (RabbitMQ connection timeouts, full sync required, ...) We were now able to reproduce the error on our Ussuri DEV cloud as well: 1) First we used strace - -p $PID_OF_NEUTRON_LINUXBRIDGE_AGENT and noticed that the data exchange on the unix socket between the rootwrap-daemon and the main process is really really slow. One could actually read line by line the read calls to the fd of the socket. 2) We then (after adding lots of log lines and other intensive manual debugging) used py-spy (https://github.com/benfred/py-spy) via "py-spy top --pid $PID" on the running neutron-linuxbridge-agent process and noticed all the CPU time (process was at 100% most of the time) was spent in msgpack/fallback.py 3) Since the issue was not observed in TRAIN we compared the msgpack version used and noticed that TRAIN was using version 0.5.6 while Ussuri upgraded this dependency to 0.6.2. 4) We then downgraded to version 0.5.6 of msgpack (ignoring the actual dependencies) --- cut --- apt policy python3-msgpack python3-msgpack: Installed: 0.6.2-1~cloud0 Candidate: 0.6.2-1~cloud0 Version table: *** 0.6.2-1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages 0.5.6-1 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status --- cut --- vs. --- cut --- apt policy python3-msgpack python3-msgpack: Installed: 0.5.6-1 Candidate: 0.6.2-1~cloud0 Version table: 0.6.2-1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages *** 0.5.6-1 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status --- cut --- and et voila: The Neutron-Linuxbridge-Agent worked just like before (building one port every few seconds) and all network ports eventually converged to ACTIVE. I could not yet spot which commit of msgpack changes (https://github.com/msgpack/msgpack-python/compare/0.5.6...v0.6.2) might have caused this issue, but I am really certain that this is a major issue for Ussuri on Ubuntu Bionic. There are "similar" issues with * https://bugs.launchpad.net/oslo.privsep/+bug/1844822 * https://bugs.launchpad.net/oslo.privsep/+bug/1896734 both related to msgpack or the size of messages exchanged. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1937261/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1937261] Re: python3-msgpack package broken due to outdated cython
python-msgpack promoted to Ussuri updates pocket. ** Changed in: cloud-archive/ussuri Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1937261 Title: python3-msgpack package broken due to outdated cython Status in Ubuntu Cloud Archive: Invalid Status in Ubuntu Cloud Archive ussuri series: Fix Released Status in neutron: New Status in oslo.privsep: New Bug description: After a successful upgrade of the control-plance from Train -> Ussuri on Ubuntu Bionic, we upgraded a first compute / network node and immediately ran into issues with Neutron: We noticed that Neutron is extremely slow in setting up and wiring the network ports, so slow it would never finish and throw all sorts of errors (RabbitMQ connection timeouts, full sync required, ...) We were now able to reproduce the error on our Ussuri DEV cloud as well: 1) First we used strace - -p $PID_OF_NEUTRON_LINUXBRIDGE_AGENT and noticed that the data exchange on the unix socket between the rootwrap-daemon and the main process is really really slow. One could actually read line by line the read calls to the fd of the socket. 2) We then (after adding lots of log lines and other intensive manual debugging) used py-spy (https://github.com/benfred/py-spy) via "py-spy top --pid $PID" on the running neutron-linuxbridge-agent process and noticed all the CPU time (process was at 100% most of the time) was spent in msgpack/fallback.py 3) Since the issue was not observed in TRAIN we compared the msgpack version used and noticed that TRAIN was using version 0.5.6 while Ussuri upgraded this dependency to 0.6.2. 4) We then downgraded to version 0.5.6 of msgpack (ignoring the actual dependencies) --- cut --- apt policy python3-msgpack python3-msgpack: Installed: 0.6.2-1~cloud0 Candidate: 0.6.2-1~cloud0 Version table: *** 0.6.2-1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages 0.5.6-1 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status --- cut --- vs. --- cut --- apt policy python3-msgpack python3-msgpack: Installed: 0.5.6-1 Candidate: 0.6.2-1~cloud0 Version table: 0.6.2-1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages *** 0.5.6-1 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status --- cut --- and et voila: The Neutron-Linuxbridge-Agent worked just like before (building one port every few seconds) and all network ports eventually converged to ACTIVE. I could not yet spot which commit of msgpack changes (https://github.com/msgpack/msgpack-python/compare/0.5.6...v0.6.2) might have caused this issue, but I am really certain that this is a major issue for Ussuri on Ubuntu Bionic. There are "similar" issues with * https://bugs.launchpad.net/oslo.privsep/+bug/1844822 * https://bugs.launchpad.net/oslo.privsep/+bug/1896734 both related to msgpack or the size of messages exchanged. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1937261/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1937261] Re: python3-msgpack package broken due to outdated cython
** Changed in: cloud-archive Status: New => Confirmed ** Also affects: cloud-archive/ussuri Importance: Undecided Status: New ** Changed in: cloud-archive Status: Confirmed => Invalid ** Changed in: cloud-archive/ussuri Status: New => Triaged ** Changed in: cloud-archive/ussuri Importance: Undecided => Medium ** No longer affects: python-msgpack (Ubuntu) ** No longer affects: python-oslo.privsep (Ubuntu) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1937261 Title: python3-msgpack package broken due to outdated cython Status in Ubuntu Cloud Archive: Invalid Status in Ubuntu Cloud Archive ussuri series: Triaged Status in neutron: New Status in oslo.privsep: New Bug description: After a successful upgrade of the control-plance from Train -> Ussuri on Ubuntu Bionic, we upgraded a first compute / network node and immediately ran into issues with Neutron: We noticed that Neutron is extremely slow in setting up and wiring the network ports, so slow it would never finish and throw all sorts of errors (RabbitMQ connection timeouts, full sync required, ...) We were now able to reproduce the error on our Ussuri DEV cloud as well: 1) First we used strace - -p $PID_OF_NEUTRON_LINUXBRIDGE_AGENT and noticed that the data exchange on the unix socket between the rootwrap-daemon and the main process is really really slow. One could actually read line by line the read calls to the fd of the socket. 2) We then (after adding lots of log lines and other intensive manual debugging) used py-spy (https://github.com/benfred/py-spy) via "py-spy top --pid $PID" on the running neutron-linuxbridge-agent process and noticed all the CPU time (process was at 100% most of the time) was spent in msgpack/fallback.py 3) Since the issue was not observed in TRAIN we compared the msgpack version used and noticed that TRAIN was using version 0.5.6 while Ussuri upgraded this dependency to 0.6.2. 4) We then downgraded to version 0.5.6 of msgpack (ignoring the actual dependencies) --- cut --- apt policy python3-msgpack python3-msgpack: Installed: 0.6.2-1~cloud0 Candidate: 0.6.2-1~cloud0 Version table: *** 0.6.2-1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages 0.5.6-1 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status --- cut --- vs. --- cut --- apt policy python3-msgpack python3-msgpack: Installed: 0.5.6-1 Candidate: 0.6.2-1~cloud0 Version table: 0.6.2-1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages *** 0.5.6-1 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status --- cut --- and et voila: The Neutron-Linuxbridge-Agent worked just like before (building one port every few seconds) and all network ports eventually converged to ACTIVE. I could not yet spot which commit of msgpack changes (https://github.com/msgpack/msgpack-python/compare/0.5.6...v0.6.2) might have caused this issue, but I am really certain that this is a major issue for Ussuri on Ubuntu Bionic. There are "similar" issues with * https://bugs.launchpad.net/oslo.privsep/+bug/1844822 * https://bugs.launchpad.net/oslo.privsep/+bug/1896734 both related to msgpack or the size of messages exchanged. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1937261/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1937261] Re: python3-msgpack package broken due to outdated cython
** Also affects: python-msgpack (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1937261 Title: python3-msgpack package broken due to outdated cython Status in Ubuntu Cloud Archive: New Status in neutron: New Status in oslo.privsep: New Status in python-msgpack package in Ubuntu: New Status in python-oslo.privsep package in Ubuntu: New Bug description: After a successful upgrade of the control-plance from Train -> Ussuri on Ubuntu Bionic, we upgraded a first compute / network node and immediately ran into issues with Neutron: We noticed that Neutron is extremely slow in setting up and wiring the network ports, so slow it would never finish and throw all sorts of errors (RabbitMQ connection timeouts, full sync required, ...) We were now able to reproduce the error on our Ussuri DEV cloud as well: 1) First we used strace - -p $PID_OF_NEUTRON_LINUXBRIDGE_AGENT and noticed that the data exchange on the unix socket between the rootwrap-daemon and the main process is really really slow. One could actually read line by line the read calls to the fd of the socket. 2) We then (after adding lots of log lines and other intensive manual debugging) used py-spy (https://github.com/benfred/py-spy) via "py-spy top --pid $PID" on the running neutron-linuxbridge-agent process and noticed all the CPU time (process was at 100% most of the time) was spent in msgpack/fallback.py 3) Since the issue was not observed in TRAIN we compared the msgpack version used and noticed that TRAIN was using version 0.5.6 while Ussuri upgraded this dependency to 0.6.2. 4) We then downgraded to version 0.5.6 of msgpack (ignoring the actual dependencies) --- cut --- apt policy python3-msgpack python3-msgpack: Installed: 0.6.2-1~cloud0 Candidate: 0.6.2-1~cloud0 Version table: *** 0.6.2-1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages 0.5.6-1 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status --- cut --- vs. --- cut --- apt policy python3-msgpack python3-msgpack: Installed: 0.5.6-1 Candidate: 0.6.2-1~cloud0 Version table: 0.6.2-1~cloud0 500 500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-updates/ussuri/main amd64 Packages *** 0.5.6-1 500 500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status --- cut --- and et voila: The Neutron-Linuxbridge-Agent worked just like before (building one port every few seconds) and all network ports eventually converged to ACTIVE. I could not yet spot which commit of msgpack changes (https://github.com/msgpack/msgpack-python/compare/0.5.6...v0.6.2) might have caused this issue, but I am really certain that this is a major issue for Ussuri on Ubuntu Bionic. There are "similar" issues with * https://bugs.launchpad.net/oslo.privsep/+bug/1844822 * https://bugs.launchpad.net/oslo.privsep/+bug/1896734 both related to msgpack or the size of messages exchanged. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1937261/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
