TLDR: Hopefully arm issues are resolved so jobs should correctly verify (they work in sandbox, production test still running)
Two different issues: JNLP connect failures: cause unconfirmed but certainly I was seeing latency issues between jenkins slave and master varying between 1ms and 2000ms. These latency issues in the past have caused issues similar issues to what we have seen this morning. I have opened a ticket with the LF about the latency spikes and also inquired about what if any latency monitoring exists to track these problems in the future. ARM-compile failure: In resolving arm-merge issues earlier this week the arm build image was rebuilt. As part of rebuilding (which is normal) ubuntu packages (including gcc) were updated to the latest released versions. There seem to be an issue with one of those updates (I’m not going to say it is gcc, only that gcc was one of the packages that were updated) that cause that panic during the build. Hopefully one of the arm knowledgable folks can work with me to try and debug and work out this problem. It is 100% reproducible after a simple apt-get upgrade so I’m optimistic that this can be resolved without too much trouble. Ill continue to watch throughout the weekend to make sure the latency problem doesn’t pop back up. thanks, Ed PS thanks for dave wallace for the extra set of eyes and thoughts during the debug On May 10, 2019, at 9:07 AM, Ed Kern via Lists.Fd.Io <ejk=cisco....@lists.fd.io<mailto:ejk=cisco....@lists.fd.io>> wrote: Two different issues currently with arm builds…. there is a failure to launch due to JNLP connect that has cropped up in the last day. But when the run does actually start we are getting these failures on master… 09:02:13 [144/1770] Building C object vlib/CMakeFiles/vlib_thunderx2t99.dir/punt_node.c.o 09:02:13 FAILED: vlib/CMakeFiles/vlib_thunderx2t99.dir/punt_node.c.o 09:02:13 ccache /usr/lib/ccache/cc -DHAVE_MEMFD_CREATE -I/w/workspace/vpp-arm-verify-master-ubuntu1804/src -I. -Iinclude -Wno-address-of-packed-member -march=armv8-a+crc -g -O2 -DFORTIFY_SOURCE=2 -fstack-protector -fPIC -Werror -fPIC -DCLIB_MARCH_VARIANT=thunderx2t99 -Wall -fno-common -march=armv8.1-a+crc+crypto -mtune=thunderx2t99 -DCLIB_N_PREFETCHES=8 -MD -MT vlib/CMakeFiles/vlib_thunderx2t99.dir/punt_node.c.o -MF vlib/CMakeFiles/vlib_thunderx2t99.dir/punt_node.c.o.d -o vlib/CMakeFiles/vlib_thunderx2t99.dir/punt_node.c.o -c /w/workspace/vpp-arm-verify-master-ubuntu1804/src/vlib/punt_node.c 09:02:13 /w/workspace/vpp-arm-verify-master-ubuntu1804/src/vlib/punt_node.c: In function ‘punt_dispatch_node_fn_thunderx2t99’: 09:02:13 /w/workspace/vpp-arm-verify-master-ubuntu1804/src/vlib/punt_node.c:280:1: internal compiler error: Segmentation fault 09:02:13 } 09:02:13 ^ 09:02:13 Please submit a full bug report, 09:02:13 with preprocessed source if appropriate. So Ill keep working on the connect issue…but even if those calm down or are worked out Im not sure things are going to pass regardless. Ed -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12984): https://lists.fd.io/g/vpp-dev/message/12984 Mute This Topic: https://lists.fd.io/mt/31577929/675649 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [e...@cisco.com<mailto:e...@cisco.com>] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#12993): https://lists.fd.io/g/vpp-dev/message/12993 Mute This Topic: https://lists.fd.io/mt/31577929/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
