Re: [vpp-dev] Strange C Dyanmic Loading Failure
Dear Billy, See https://gerrit.fd.io/r/#/c/5918. Exactly as described: multiple spurious client link dependencies against libvlib.so. Tested on FC-25, Ubuntu. It’ll take a while to verify the patch. In the meanwhile, please kick the tires. HTH… Dave From: Billy McFall [mailto:bmcf...@redhat.com] Sent: Wednesday, March 29, 2017 10:46 AM To: Dave Barach (dbarach) Cc: Thomas F Herbert Subject: Re: [vpp-dev] Strange C Dyanmic Loading Failure Thanks, let me know what I can do to help. On Wed, Mar 29, 2017 at 10:43 AM, Dave Barach (dbarach) mailto:dbar...@cisco.com>> wrote: Net of considerable fucking around, I managed to repro the problem on FC-25. Sure enough, vpp_api_test is linked against libvlib.so. It should not be. Let me see if I can work out why the set of libraries is wrong. Thanks… Dave From: Billy McFall [mailto:bmcf...@redhat.com<mailto:bmcf...@redhat.com>] Sent: Wednesday, March 29, 2017 10:39 AM To: Dave Barach (dbarach) mailto:dbar...@cisco.com>>; Thomas F Herbert mailto:therb...@redhat.com>> Subject: Re: [vpp-dev] Strange C Dyanmic Loading Failure cc Tom, he wanted to be in the loop on this, Sorry, forgot about the dependencies. In vpp/Makefile, there are some hardcoded versions of libraries that are older than what are in the Fedora 25 repo. I just delete the versions if I need to do a 'make install-dep'. Tom knows about the issue. I would just remove the versions myself, but I don't know why they were put in there in the first place, so I don't want to break an older distro I don't test with. git diff Makefile diff --git a/Makefile b/Makefile index bdf20e8..b92251a 100644 --- a/Makefile +++ b/Makefile @ -190,7 +190,7 @@ else ifneq ("$(wildcard /etc/redhat-release)","") @sudo -E yum groupinstall $(CONFIRM) $(RPM_DEPENDS_GROUPS) @sudo -E yum install $(CONFIRM) $(RPM_DEPENDS) @sudo -E yum install $(CONFIRM) --enablerepo=epel $(EPEL_DEPENDS) - @sudo -E debuginfo-install $(CONFIRM) glibc-2.17-106.el7_2.4.x86_64 openssl-libs-1.0.1e-51.el7_2.4.x86_64 zlib-1.2.7-15.el7.x86_64 + @sudo -E debuginfo-install $(CONFIRM) glibc openssl-libs zlib As far as building. Send the the output and I'll see if I typically see the errors you are seeing. To build, I use: cd vpp; make install-dep; make bootstrap; make build-release; make pkg-rpm If I want to do a debug build, I edit the vpp/Makefile as follows (Tom might have a cleaner way): git diff Makefile diff --git a/Makefile b/Makefile index bdf20e8..b92251a 100644 --- a/Makefile +++ b/Makefile @@ -225,7 +225,7 @@ wipe: wipedist $(BR)/.bootstrap.ok rebuild: wipe build build-release: $(BR)/.bootstrap.ok - $(call make,$(PLATFORM),vpp-install) + $(call make,$(PLATFORM)_debug,vpp-install) wipe-release: $(BR)/.bootstrap.ok $(call make,$(PLATFORM),vpp-wipe) @@ -342,7 +342,7 @@ pkg-deb: $(call make,$(PLATFORM),install-deb) pkg-rpm: dist - $(call make,$(PLATFORM),install-rpm) + $(call make,$(PLATFORM)_debug,install-rpm) Any thoughts on why the libvlib.so is still linked to vpp_api_test on CentOS but it doesn't crash? Billy On Wed, Mar 29, 2017 at 10:22 AM, Dave Barach (dbarach) mailto:dbar...@cisco.com>> wrote: So I set up a Fedora 25 VM. Aside from having to manually install a whole bunch of dependencies, what’s the exact trick to build and install RPMs in that environment? $ make PLATFORM=vpp TAG=vpp_debug install-rpm Fails with some annoying RPM complaint that I don’t understand. Thanks… Dave From: Billy McFall [mailto:bmcf...@redhat.com<mailto:bmcf...@redhat.com>] Sent: Wednesday, March 29, 2017 9:37 AM To: Dave Barach (dbarach) mailto:dbar...@cisco.com>> Subject: Re: [vpp-dev] Strange C Dyanmic Loading Failure My CentOS VM with a clean sandbox from an hour ago built, loaded and ran fine. I was able to run 'vppctl show interface' without a crash. However, libvlib.so is still linked to vpp_api_test. $ ldd /usr/bin/vpp_api_test | grep libvlib.so libvlib.so.0 => /lib64/libvlib.so.0 (0x7fb0179a8000) Billy On Wed, Mar 29, 2017 at 9:03 AM, Billy McFall mailto:bmcf...@redhat.com>> wrote: If you want to wait on the Fedora 25 until I test CentOS 7, that's fine with me. On Wed, Mar 29, 2017 at 8:57 AM, Billy McFall mailto:bmcf...@redhat.com>> wrote: My CentOS server and CentOS VM both have libvlib.so linked, but they are also older builds, I will create a clean sandbox in my CentOS VM and see how it behaves. On Wed, Mar 29, 2017 at 8:51 AM, Dave Barach (dbarach) mailto:dbar...@cisco.com>> wrote: Here’s what I see from ldd on an Ubuntu 16.04 system. Note the conspicuous lack of libvlib.so...: root@wasa-ucs-12:/scratch/dbarach/vpp-tcp/build-root# ldd /usr/bin/vpp_api_test linux-vdso.so.1 => (0x77ffd000) libvlibmemoryclient.so.0 => /usr/lib/x86_6
Re: [vpp-dev] Strange C Dyanmic Loading Failure
On Mon, Mar 27, 2017 at 9:27 AM, Adrian Daniel Calianu < adrian.cali...@gmail.com> wrote: > Hi all, > > I have the same issue. After I built the vpp from sources(master branch) > and run vppctl I see a segmentation fault. This looks to be caused by the > fact that the vlib_mains from init functions of dynamic linked libraries is > 0x0. > Following patch will fix the problem: > --- a/src/vlib/global_funcs.h > 19 +++ b/src/vlib/global_funcs.h > 20 @@ -23,7 +23,7 @@ always_inline vlib_main_t * > 21 vlib_get_main (void) > 22 { > 23vlib_main_t *vm; > 24 - vm = vlib_mains[os_get_cpu_number ()]; > 25 + vm = vlib_mains ? vlib_mains[os_get_cpu_number ()] : > &vlib_global_main; > 26ASSERT (vm); > 27return vm; > 28 } > > This is like a part of patch did in commit: > 80f54e20270ed0628ee725e3e3c515731a0188f2 > need to be reverted. > > Is other fix of this issue? > > /Adrian > For the record, my issue was PEBCAK. I needed to add -lvlib to my link line. Oh yeah. jdl ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
Re: [vpp-dev] Strange C Dyanmic Loading Failure
Guys, We’re not going to revert that patch. I strongly suspect that the problem is due to stale bits installed on the host. Please check your system against this result: $ ldd /usr/bin/vpp_api_test linux-vdso.so.1 => (0x7fff69357000) libvlibmemoryclient.so.0 => /usr/lib/x86_64-linux-gnu/libvlibmemoryclient.so.0 (0x7f38f79c8000) libsvm.so.0 => /usr/lib/x86_64-linux-gnu/libsvm.so.0 (0x7f38f77ab000) libvatplugin.so.0 => /usr/lib/x86_64-linux-gnu/libvatplugin.so.0 (0x7f38f75a7000) libvppinfra.so.0 => /usr/lib/x86_64-linux-gnu/libvppinfra.so.0 (0x7f38f72b9000) libvlibapi.so.0 => /usr/lib/x86_64-linux-gnu/libvlibapi.so.0 (0x7f38f70a3000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x7f38f6e85000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x7f38f6c81000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7f38f68b8000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x7f38f66af000) /lib64/ld-linux-x86-64.so.2 (0x564972b0c000) If you see that vpp_api_test is linked against libvlib.so, I guarantee that it will crash as described. Thanks… Dave From: Adrian Daniel Calianu [mailto:adrian.cali...@gmail.com] Sent: Monday, March 27, 2017 10:27 AM To: Jon Loeliger Cc: vpp-dev ; Dave Barach (dbarach) Subject: Re: [vpp-dev] Strange C Dyanmic Loading Failure Hi all, I have the same issue. After I built the vpp from sources(master branch) and run vppctl I see a segmentation fault. This looks to be caused by the fact that the vlib_mains from init functions of dynamic linked libraries is 0x0. Following patch will fix the problem: --- a/src/vlib/global_funcs.h 19 +++ b/src/vlib/global_funcs.h 20 @@ -23,7 +23,7 @@ always_inline vlib_main_t * 21 vlib_get_main (void) 22 { 23vlib_main_t *vm; 24 - vm = vlib_mains[os_get_cpu_number ()]; 25 + vm = vlib_mains ? vlib_mains[os_get_cpu_number ()] : &vlib_global_main; 26ASSERT (vm); 27return vm; 28 } This is like a part of patch did in commit: 80f54e20270ed0628ee725e3e3c515731a0188f2 need to be reverted. Is other fix of this issue? /Adrian On Tue, Mar 21, 2017 at 4:34 AM, Jon Loeliger mailto:j...@netgate.com>> wrote: On Mon, Mar 20, 2017 at 9:18 PM, Jon Loeliger mailto:j...@netgate.com>> wrote: Dave, With the advent of a compiling tree (hooray!), I've updated, rebuilt, installed, and tried running all the C libraries and API pieces again. Sadly, things are not working. Very early in the bring-up of a C executable that is linked to VPP's libraries (vlibmemoryclient vlibapi svm vppinfra), it goes into the weeds trying to load some statically configured CLI initialization functions dynamically: 80f54e20270ed0628ee725e3e3c515731a0188f2 Author: Dave Barach mailto:d...@barachs.net>> Date: Wed Mar 8 19:08:56 2017 -0500 vlib_mains == 0 special cases be gone Clean up spurious binary API client link dependency on libvlib.so, which managed to hide behind vlib_mains == 0 checks reached by VLIB_xxx_FUNCTION macros. Change-Id: I5df1f8ab07dca1944250e643ccf06e60a8462325 Signed-off-by: Dave Barach mailto:d...@barachs.net>> Also, if one simply 'git reverts' this one commit on the current top of tree, things resume working again. HTH, jdl ___ vpp-dev mailing list vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io> https://lists.fd.io/mailman/listinfo/vpp-dev ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
Re: [vpp-dev] Strange C Dyanmic Loading Failure
Hi all, I have the same issue. After I built the vpp from sources(master branch) and run vppctl I see a segmentation fault. This looks to be caused by the fact that the vlib_mains from init functions of dynamic linked libraries is 0x0. Following patch will fix the problem: --- a/src/vlib/global_funcs.h 19 +++ b/src/vlib/global_funcs.h 20 @@ -23,7 +23,7 @@ always_inline vlib_main_t * 21 vlib_get_main (void) 22 { 23vlib_main_t *vm; 24 - vm = vlib_mains[os_get_cpu_number ()]; 25 + vm = vlib_mains ? vlib_mains[os_get_cpu_number ()] : &vlib_global_main; 26ASSERT (vm); 27return vm; 28 } This is like a part of patch did in commit: 80f54e20270ed0628ee725e3e3c515731a0188f2 need to be reverted. Is other fix of this issue? /Adrian On Tue, Mar 21, 2017 at 4:34 AM, Jon Loeliger wrote: > On Mon, Mar 20, 2017 at 9:18 PM, Jon Loeliger wrote: > >> Dave, >> >> With the advent of a compiling tree (hooray!), I've updated, rebuilt, >> installed, and tried running all the C libraries and API pieces again. >> Sadly, things are not working. >> >> Very early in the bring-up of a C executable that is linked to VPP's >> libraries >> (vlibmemoryclient vlibapi svm vppinfra), it goes into the weeds trying to >> load some statically configured CLI initialization functions dynamically: >> >> > >> 80f54e20270ed0628ee725e3e3c515731a0188f2 >> Author: Dave Barach >> Date: Wed Mar 8 19:08:56 2017 -0500 >> >> vlib_mains == 0 special cases be gone >> >> Clean up spurious binary API client link dependency on libvlib.so, >> which managed to hide behind vlib_mains == 0 checks reached by >> VLIB_xxx_FUNCTION macros. >> >> Change-Id: I5df1f8ab07dca1944250e643ccf06e60a8462325 >> Signed-off-by: Dave Barach >> > > Also, if one simply 'git reverts' this one commit on the current top > of tree, things resume working again. > > HTH, > jdl > > > ___ > vpp-dev mailing list > vpp-dev@lists.fd.io > https://lists.fd.io/mailman/listinfo/vpp-dev > ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
Re: [vpp-dev] Strange C Dyanmic Loading Failure
On Mon, Mar 20, 2017 at 9:18 PM, Jon Loeliger wrote: > Dave, > > With the advent of a compiling tree (hooray!), I've updated, rebuilt, > installed, and tried running all the C libraries and API pieces again. > Sadly, things are not working. > > Very early in the bring-up of a C executable that is linked to VPP's > libraries > (vlibmemoryclient vlibapi svm vppinfra), it goes into the weeds trying to > load some statically configured CLI initialization functions dynamically: > > > 80f54e20270ed0628ee725e3e3c515731a0188f2 > Author: Dave Barach > Date: Wed Mar 8 19:08:56 2017 -0500 > > vlib_mains == 0 special cases be gone > > Clean up spurious binary API client link dependency on libvlib.so, > which managed to hide behind vlib_mains == 0 checks reached by > VLIB_xxx_FUNCTION macros. > > Change-Id: I5df1f8ab07dca1944250e643ccf06e60a8462325 > Signed-off-by: Dave Barach > Also, if one simply 'git reverts' this one commit on the current top of tree, things resume working again. HTH, jdl ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
[vpp-dev] Strange C Dyanmic Loading Failure
Dave, With the advent of a compiling tree (hooray!), I've updated, rebuilt, installed, and tried running all the C libraries and API pieces again. Sadly, things are not working. Very early in the bring-up of a C executable that is linked to VPP's libraries (vlibmemoryclient vlibapi svm vppinfra), it goes into the weeds trying to load some statically configured CLI initialization functions dynamically: That error looks like this: Program received signal SIGSEGV, Segmentation fault. __vlib_cli_command_registration_cli_show_api_histogram_command () at //vpp/src/vlibmemory/memory_vlib.c:727 727 VLIB_CLI_COMMAND (cli_show_api_histogram_command, static) = { Missing separate debuginfos, use: debuginfo-install clixon-3.2-0.x86_64 (gdb) where #0 __vlib_cli_command_registration_cli_show_api_histogram_command () at //vpp/src/vlibmemory/memory_vlib.c:727 #1 0x77dec1e3 in call_init (env=0x7fffe3d0, argv=0x7fffe398, argc=6, l=) at dl-init.c:82 #2 _dl_init (main_map=main_map@entry=0x760790, argc=6, argv=0x7fffe398, env=0x7fffe3d0) at dl-init.c:131 #3 0x77df08f6 in dl_open_worker (a=a@entry=0x7fffde28) at dl-open.c:560 #4 0x77debff4 in _dl_catch_error ( objname=objname@entry=0x7fffde18, errstring=errstring@entry=0x7fffde20, mallocedp=mallocedp@entry=0x7fffde10, operate=operate@entry=0x77df0540 , args=args@entry=0x7fffde28) at dl-error.c:177 #5 0x77deffeb in _dl_open ( file=0x77ff3f20 "/../vpp_backend.so", mode=-2147483646, caller_dlopen=, nsid=-2, argc=6, argv=0x7fffe398, env=0x7fffe3d0) at dl-open.c:650 As we are not using any aspect of the VPP project's actual CLI code, there is no way we should be loading anything like a CLI command. I suspect that this init data structure (line 727 of src/vlibmemory/memory_vlib.c) is incorrectly present in the API library due to some *other* symbol in this object file being referenced and dragging in the CLI init piece accidentally. Here is line 727: VLIB_CLI_COMMAND (cli_show_api_histogram_command, static) = { .path = "show api histogram", .short_help = "show api histogram", .function = vl_api_show_histogram_command, }; I have git bisected the introduction of this bad behavior to this commit: 80f54e20270ed0628ee725e3e3c515731a0188f2 Author: Dave Barach Date: Wed Mar 8 19:08:56 2017 -0500 vlib_mains == 0 special cases be gone Clean up spurious binary API client link dependency on libvlib.so, which managed to hide behind vlib_mains == 0 checks reached by VLIB_xxx_FUNCTION macros. Change-Id: I5df1f8ab07dca1944250e643ccf06e60a8462325 Signed-off-by: Dave Barach If no one beats me to it, I'll try to isolate it a bit more tomorrow. Thanks, jdl ___ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev