Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-17 Thread Waldek Kozaczuk
I have opened 3 issues to track all the problems we discussed/identified in this thread: - https://github.com/cloudius-systems/osv/issues/1067 - https://github.com/cloudius-systems/osv/issues/1066 - https://github.com/cloudius-systems/osv/issues/1065 On Monday, December 16, 2019 at

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-16 Thread zhiting zhu
Have you guys considered running elf fuzzing: https://github.com/IOActive/Melkor_ELF_Fuzzer? This tool pops up when I'm searching for elf standard testing. It doesn't test TLS handing though. On Mon, Dec 16, 2019 at 1:17 PM Waldek Kozaczuk wrote: > Regarding local STT_TLS based on this - > https

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-16 Thread Waldek Kozaczuk
Regarding local STT_TLS based on this - https://www.openwall.com/lists/musl/2019/05/24/1 - here is the simple reproducer: cat vers { global: get_tls_var; local: *; }; cat dso.c __thread int tlsvar = 11; int get_tls_var(void) { return tlsvar; } cat main.c #include int get_tls_var(void)

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-16 Thread Waldek Kozaczuk
It could be that musl also does not handle LOCAL STT_TLS well when ELF linked with gold linker (and glibc does). Here is a thread I have found that seems to be applicable to us as well - https://www.openwall.com/lists/musl/2019/05/25/2. This pseudo-patch (includes yours so you might need to a

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-15 Thread Waldek Kozaczuk
Copying to the list. On Sun, Dec 15, 2019 at 20:38 Waldek Kozaczuk wrote: > It might be similar (or have same root cause) as the issue with > gCurrentThreadInfo reported here - > https://groups.google.com/forum/m/#!topic/osv-dev/k69cHw7qvTg. In both > cases the symbols are TLS but in other case

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-15 Thread zhiting zhu
I comment out version_symtab related code and it's still having the same problem. It's still failing hash comparison. Maybe I miss some other places? The hash comparison happens before version checking. Best, Zhiting On Sun, Dec 15, 2019 at 6:31 PM Waldek Kozaczuk wrote: > Try to disable th

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-15 Thread Waldek Kozaczuk
Try to disable the version check - it might be the other bug when elf is referencing its own old symbol. Sent from my iPhone > On Dec 15, 2019, at 19:09, zhiting zhu wrote: > >  > I encounter another failed to lookup symbol error. It's > > 2813: 0058 8 TLS LOCAL DEFAUL

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-15 Thread zhiting zhu
I encounter another failed to lookup symbol error. It's 2813: 0058 8 TLS LOCAL DEFAULT 20 _ZN9grpc_core7ExecCtx9exe The symbol is defined in tensorflow/python/_pywrap_tensorflow_internal.so. I check symbol visibility. It's visible. I add some print out in lookup_symbol_gnu.

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-15 Thread zhiting zhu
Thanks for the explanation. This patch seems does the job: Author: Zhiting Zhu Date: Sun Dec 15 00:57:36 2019 -0600 handling STT_SECTION symbol Signed-off-by: Zhiting Zhu diff --git a/core/elf.cc b/core/elf.cc index e2b5b667..a5be192b 100644 --- a/core/elf.cc +++ b/core/elf.cc @@ -

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-14 Thread Waldek Kozaczuk
After some trial and error, I did manage to reproduce your issue. While doing that I also got to understand other reported earlier issue with numpy and blas_memory_alloc symbol. First about it. After adding more debug here is what I saw and it makes all self-explanatory: ELF [tid:253, /lib/pyt

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-14 Thread zhiting zhu
I think the following would work. python3 -m pip install tensorflow==1.14 (I think google's official binary should work on your machine. I check that they have the same problem). Replace the GET file in apps/python3x with the one attached. ./scripts/build fs_size_mb=1024 image=python3x ./script

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-14 Thread Waldek Kozaczuk
Or dockerfile so that I can easily recreate the image. On Fri, Dec 13, 2019 at 20:31 Waldek Kozaczuk wrote: > I downloaded the wheel file but I get this error (I am using Ubuntu 19.10): > > pip3 install ./tensorflow-1.14.1-cp36-cp36m-linux_x86_64.whl > tensorflow-1.14.1-cp36-cp36m-linux_x86_64.w

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-13 Thread Waldek Kozaczuk
I downloaded the wheel file but I get this error (I am using Ubuntu 19.10): pip3 install ./tensorflow-1.14.1-cp36-cp36m-linux_x86_64.whl tensorflow-1.14.1-cp36-cp36m-linux_x86_64.whl is not a supported wheel on this platform. Could you please send us full recipe that would build the app? Ideall

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-13 Thread zhiting zhu
No. It's Num:Value Size TypeBind Vis Ndx Name 1: 0affa7f8 0 SECTION LOCAL DEFAULT 21 It's not hidden. Here's the new link: https://send.firefox.com/download/9a8bf3fa2909635f/#0ZecrR7UJwspr743vNBo6A to the file. On Fri, Dec 13, 2019 at 4:44 PM Waldek Koza

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-13 Thread Waldek Kozaczuk
I am not sure but this issue is similar to what I encountered when dealing with dotnet: readelf -s libcoreclr.so | grep gCurrentThreadInfo readelf: Warning: local symbol 31 found at index >= .dynsym's sh_info value of 1 31: 24 TLS LOCAL HIDDEN19 gCurrentThreadIn

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-13 Thread zhiting zhu
I was reading the print wrong. I think the problem is this one: 0b23ca60 00010010 R_X86_64_DTPMOD64 0aee9f58 .tbss + 0 The sym is 1 but in this case, .tbss is still in the same shared object but in arch_relocate_rela, it's going to the else branch that handles the variable is loc

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-11 Thread zhiting zhu
I get rid of the __VA_OPT and replace the __VA_ARGS__ with ##__VA_ARGS__. ## seems to be the extension from GNU that does similar things. Here's the link of the TensorFlow that I compile myself: https://send.firefox.com/download/68d2a81e2cacdafb/#ui_lMMYAU9UW9e6Fd1VG7w Install it locally and buil

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-11 Thread Waldek Kozaczuk
I wonder if that is related to a similar issue as described here - https://groups.google.com/forum/#!topic/osv-dev/k69cHw7qvTg. I will try to fix and apply this debug patch to master so it makes easier to debug it. Meanwhile, can you provide a reproducing test? On Wednesday, December 11, 2019

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-11 Thread zhiting zhu
After tracing inside arch_relocate_rela, it's failed in case R_X86_64_DTPMOD64. I print out the name index of that symbol and it's 0. I use readelf -a to check the .rela.dyn section. There's a line that doesn't have Sym. Value and Sym.Name + Addend is 0. I think I'm failing at that line. Offset

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-11 Thread zhiting zhu
Thanks for the debug patch. When I apply it, g++ complains about "core/elf.cc:36:118: error: expected ‘)’ before ‘__VA_OPT__’" I think __VA_OPT__ is only added to c++2a according to https://gcc.gnu.org/onlinedocs/cpp/Variadic-Macros.html. Gcc 7.4.0 on Ubuntu cannot compile that. On Tue, Dec 10, 20

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-10 Thread Waldek Kozaczuk
You may want to try to apply this patch - https://groups.google.com/forum/#!topic/osv-dev/LbnnY2Kcmak - that should provide many useful debug printouts. There is another patch I have sent that fixes the versioned self-lookup problem - https://groups.google.com/forum/#!topic/osv-dev/d56plMGXi6E.

Re: [osv-dev] how to find missing symbols if it doesn't print the name

2019-12-10 Thread Nadav Har'El
On Mon, Dec 9, 2019 at 10:51 PM zhiting zhu wrote: > Hey, > > I'm encountering this when I'm using some tensorflow functions: > > /lib/python3.6/tensorflow/python/_pywrap_tensorflow_internal.so: failed > looking up symbol > This is interesting, because the "failed looking up symbol" message is a

[osv-dev] how to find missing symbols if it doesn't print the name

2019-12-09 Thread zhiting zhu
Hey, I'm encountering this when I'm using some tensorflow functions: /lib/python3.6/tensorflow/python/_pywrap_tensorflow_internal.so: failed looking up symbol [backtrace] 0x403442a7 0x40397dce 0x4033eed4 0x40341d27 0x40345623 , std::allocator >, st