I encounter another failed to lookup symbol error. It's
2813: 0000000000000058 8 TLS LOCAL DEFAULT 20
_ZN9grpc_core7ExecCtx9exe
The symbol is defined in tensorflow/python/_pywrap_tensorflow_internal.so.
I check symbol visibility. It's visible. I add some print out in
lookup_symbol_gnu. It seems it can't find the symbol in that so despite it
exists. I look at the checks and all the hash comparison failed (the
following line).
if ((chains[idx] & ~1) != (hashval & ~1)) {
I'm trying to compare with the musl code:
https://github.com/ifduyue/musl/blob/79f653c6bc2881dd6855299c908a442f56cb7c2b/ldso/dynlink.c#L230-L249
I don't find that much of a difference. I check the hash calculation.
gnu_hash in musl and dl_new_hash outputs the same number for that symbol
name.
Best,
Zhiting
On Sun, Dec 15, 2019 at 12:51 PM zhiting zhu <zhiti...@cs.utexas.edu> wrote:
> Thanks for the explanation.
>
> This patch seems does the job:
>
> Author: Zhiting Zhu <zhiti...@cs.utexas.edu>
> Date: Sun Dec 15 00:57:36 2019 -0600
>
> handling STT_SECTION symbol
>
> Signed-off-by: Zhiting Zhu <zhiti...@cs.utexas.edu>
>
> diff --git a/core/elf.cc b/core/elf.cc
> index e2b5b667..a5be192b 100644
> --- a/core/elf.cc
> +++ b/core/elf.cc
> @@ -641,6 +641,10 @@ symbol_module object::symbol(unsigned idx, bool
> ignore_missing)
> auto symtab = dynamic_ptr<Elf64_Sym>(DT_SYMTAB);
> assert(dynamic_val(DT_SYMENT) == sizeof(Elf64_Sym));
> auto sym = &symtab[idx];
> + auto type = sym->st_info & 15;
> + if (type == STT_SECTION) {
> + return symbol_module(sym, this);
> + }
> auto nameidx = sym->st_name;
> auto name = dynamic_ptr<const char>(DT_STRTAB) + nameidx;
> auto ret = _prog.lookup(name);
>
> For symbol visibility, would the patch you mentioned previously(
> https://groups.google.com/forum/#!topic/osv-dev/d56plMGXi6E) fix the
> problem?
>
> Best,
> Zhiting
>
> On Sun, Dec 15, 2019 at 12:02 AM Waldek Kozaczuk <jwkozac...@gmail.com>
> wrote:
>
>> After some trial and error, I did manage to reproduce your issue. While
>> doing that I also got to understand other reported earlier issue with numpy
>> and blas_memory_alloc symbol. First about it. After adding more debug here
>> is what I saw and it makes all self-explanatory:
>>
>> ELF [tid:253, /lib/python3.7/site-packages/numpy/.libs/
>> libopenblasp-r0-34a18dc3.3.7.so]: setprivate set to: 1
>> ELF [tid:253, /lib/python3.7/site-packages/numpy/.libs/
>> libopenblasp-r0-34a18dc3.3.7.so]: Executing DT_INIT function
>> ELF [tid:253, /lib/python3.7/site-packages/numpy/.libs/
>> libopenblasp-r0-34a18dc3.3.7.so]: Finished executing DT_INIT function
>> ELF [tid:253, /lib/python3.7/site-packages/numpy/.libs/
>> libopenblasp-r0-34a18dc3.3.7.so]: Executing 2 DT_INIT_ARRAYSZ functions
>> ELF [tid:254, /libvdso.so]: Not found in /libvdso.so!
>> ELF [tid:254, /usr/lib/libexpat.so.1]: Not found in
>> /usr/lib/libexpat.so.1!
>> ELF [tid:254, /usr/lib/libz.so.1]: Not found in /usr/lib/libz.so.1!
>> ELF [tid:254, /lib/python3.7/lib-dynload/
>> readline.cpython-37m-x86_64-linux-gnu.so]: Not found in
>> /lib/python3.7/lib-dynload/readline.cpython-37m-x86_64-linux-gnu.so!
>> ELF [tid:254, /usr/lib/libtinfo.so.6]: Not found in
>> /usr/lib/libtinfo.so.6!
>> ELF [tid:254, /lib/python3.7/site-packages/numpy/core/_
>> multiarray_umath.cpython-37m-x86_64-linux-gnu.so]: blas_memory_alloc
>> _visibility: 253, visible: 0!
>> ELF [tid:254, /lib/python3.7/site-packages/numpy/.libs/
>> libopenblasp-r0-34a18dc3.3.7.so]: blas_memory_alloc _visibility: 253,
>> visible: 0!
>> ELF [tid:254, ]: Not found in !
>> /lib/python3.7/site-packages/numpy/.libs/libopenblasp-r0-34a18dc3.3.7.so:
>> failed looking up symbol blas_memory_alloc
>>
>> [backtrace]
>> 0x0000000040364205 <elf::object::symbol(unsigned int, bool)+1637>
>> 0x0000000040364306 <elf::object::resolve_pltgot(unsigned int)+134>
>> 0x000000004036462c <elf_resolve_pltgot+60>
>> 0x00000000403ad61f <???+1077597727>
>> 0x0000200000780f6f <???+7868271>
>> 0x0000000000000003 <???+3>
>>
>> As you can see the symbol blas_memory_alloc is being looked up by one of
>> DT_INIT_ARRAYSZ functions but in another thread - 254 - than when the
>> function was actually called. Our "private/visible" mechanism hides this
>> symbol from other threads. We probably should check if the object is
>> looking for its own symbol or fix it in another way.
>>
>> Your newest problem I think is related to the fact that this symbol is
>> STT_SECTION and we are not handling it right. Here is what I found in one
>> of the blogs:
>>
>> "*STT_SECTION*When I first started learning about ELF, and someone would
>> say something about "section symbols", I thought they meant a symbol from
>> some given section. That's not it though: A section symbol is a symbol that
>> is used to refer to the section itself. They are used mainly when
>> performing relocations, which are often specified in the form of "modify
>> the value at offset XXX relative to the start of section YYY"."
>>
>> Here is related code in musl dynamic linker I think -
>> https://github.com/ifduyue/musl/blob/79f653c6bc2881dd6855299c908a442f56cb7c2b/ldso/dynlink.c#L346-L371
>> (look
>> for STT_SECTION if).
>>
>> I think we should detect if the relocated symbol is of type STT_SECTION
>> and change the handling of R_X86_64_DTPMOD64 and NOT lookup the symbol in
>> other modules but instead use the module index of the object whose symbol
>> is being relocated.
>>
>> On Saturday, December 14, 2019 at 10:46:36 AM UTC-5, zhiting zhu wrote:
>>>
>>> I think the following would work.
>>>
>>> python3 -m pip install tensorflow==1.14 (I think google's official
>>> binary should work on your machine. I check that they have the same
>>> problem).
>>>
>>> Replace the GET file in apps/python3x with the one attached.
>>>
>>> ./scripts/build fs_size_mb=1024 image=python3x
>>>
>>> ./scripts/run.py -r
>>>
>>> In the python cli, type "import tensorflow".
>>>
>>> Best,
>>> Zhiting
>>>
>>>
>>>
>>> On Sat, Dec 14, 2019 at 8:23 AM Waldek Kozaczuk <jwkoz...@gmail.com>
>>> wrote:
>>>
>>>> Or dockerfile so that I can easily recreate the image.
>>>>
>>>> On Fri, Dec 13, 2019 at 20:31 Waldek Kozaczuk <jwkoz...@gmail.com>
>>>> wrote:
>>>>
>>>>> I downloaded the wheel file but I get this error (I am using Ubuntu
>>>>> 19.10):
>>>>>
>>>>> pip3 install ./tensorflow-1.14.1-cp36-cp36m-linux_x86_64.whl
>>>>> tensorflow-1.14.1-cp36-cp36m-linux_x86_64.whl is not a supported wheel
>>>>> on this platform.
>>>>>
>>>>> Could you please send us full recipe that would build the app? Ideally
>>>>> an "app" just like any we have under osv-apps repo so that I can simply
>>>>> run:
>>>>> ./scripts/buils image=????
>>>>>
>>>>> and then run it.
>>>>>
>>>>> Thanks,
>>>>> Waldek
>>>>>
>>>>> On Friday, December 13, 2019 at 5:52:03 PM UTC-5, zhiting zhu wrote:
>>>>>>
>>>>>> No. It's
>>>>>> Num: Value Size Type Bind Vis Ndx Name
>>>>>> 1: 000000000affa7f8 0 SECTION LOCAL DEFAULT 21
>>>>>>
>>>>>> It's not hidden.
>>>>>>
>>>>>> Here's the new link:
>>>>>> https://send.firefox.com/download/9a8bf3fa2909635f/#0ZecrR7UJwspr743vNBo6A
>>>>>> to the file.
>>>>>>
>>>>>> On Fri, Dec 13, 2019 at 4:44 PM Waldek Kozaczuk <jwkoz...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>> I am not sure but this issue is similar to what I encountered when
>>>>>>> dealing with dotnet:
>>>>>>>
>>>>>>> readelf -s libcoreclr.so | grep gCurrentThreadInfo
>>>>>>> readelf: Warning: local symbol 31 found at index >= .dynsym's
>>>>>>> sh_info value of 1
>>>>>>> 31: 0000000000000000 24 TLS LOCAL HIDDEN 19
>>>>>>> gCurrentThreadInfo
>>>>>>> 9799: 0000000000000000 24 TLS LOCAL HIDDEN 19
>>>>>>> gCurrentThreadInfo
>>>>>>>
>>>>>>> When you use readelf does it show it as a hidden, local TLS symbol?
>>>>>>>
>>>>>>> That link with compiles expired. Can you upload it somewhere again?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Friday, December 13, 2019 at 1:16:21 PM UTC-5, zhiting zhu wrote:
>>>>>>>>
>>>>>>>> I was reading the print wrong.
>>>>>>>> I think the problem is this one:
>>>>>>>>
>>>>>>>> 00000b23ca60 000100000010 R_X86_64_DTPMOD64 000000000aee9f58 .tbss
>>>>>>>> + 0
>>>>>>>>
>>>>>>>> The sym is 1 but in this case, .tbss is still in the same shared
>>>>>>>> object but in arch_relocate_rela, it's going to the else branch that
>>>>>>>> handles the variable is located in DIFFERENT shared object that the
>>>>>>>> caller.
>>>>>>>>
>>>>>>>> On Wed, Dec 11, 2019 at 3:12 PM zhiting zhu <zhit...@cs.utexas.edu>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I get rid of the __VA_OPT and replace the __VA_ARGS__ with
>>>>>>>>> ##__VA_ARGS__. ## seems to be the extension from GNU that does similar
>>>>>>>>> things.
>>>>>>>>>
>>>>>>>>> Here's the link of the TensorFlow that I compile myself:
>>>>>>>>> https://send.firefox.com/download/68d2a81e2cacdafb/#ui_lMMYAU9UW9e6Fd1VG7w
>>>>>>>>>
>>>>>>>>> Install it locally and build an osv vm that includes all the
>>>>>>>>> dependencies:
>>>>>>>>> "tensorflow grpc google _cffi_backend past future \
>>>>>>>>> absl wrapt gast astor termcolor numpy unittest libfuturize \
>>>>>>>>> keras_applications keras_preprocessing tensorflow_estimator \
>>>>>>>>> tensorboard".
>>>>>>>>>
>>>>>>>>> Then just run "import tensorflow" in python shell, it should show
>>>>>>>>> you the error I'm looking at.
>>>>>>>>>
>>>>>>>>> On Wed, Dec 11, 2019 at 2:55 PM Waldek Kozaczuk <
>>>>>>>>> jwkoz...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I wonder if that is related to a similar issue as described here
>>>>>>>>>> - https://groups.google.com/forum/#!topic/osv-dev/k69cHw7qvTg.
>>>>>>>>>>
>>>>>>>>>> I will try to fix and apply this debug patch to master so it
>>>>>>>>>> makes easier to debug it.
>>>>>>>>>>
>>>>>>>>>> Meanwhile, can you provide a reproducing test?
>>>>>>>>>>
>>>>>>>>>> On Wednesday, December 11, 2019 at 3:49:07 PM UTC-5, zhiting zhu
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> After tracing inside arch_relocate_rela, it's failed in case
>>>>>>>>>>> R_X86_64_DTPMOD64.
>>>>>>>>>>>
>>>>>>>>>>> I print out the name index of that symbol and it's 0. I use
>>>>>>>>>>> readelf -a to check the .rela.dyn section.
>>>>>>>>>>> There's a line that doesn't have Sym. Value and Sym.Name +
>>>>>>>>>>> Addend is 0. I think I'm failing at that line.
>>>>>>>>>>>
>>>>>>>>>>> Offset Info Type Sym. Value
>>>>>>>>>>> Sym. Name + Addend
>>>>>>>>>>> *00000b202f90 000000000010 R_X86_64_DTPMOD64
>>>>>>>>>>> 0*
>>>>>>>>>>> 00000b23ca60 000100000010 R_X86_64_DTPMOD64 000000000aee9f58
>>>>>>>>>>> .tbss + 0
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 11, 2019 at 10:36 AM zhiting zhu <
>>>>>>>>>>> zhit...@cs.utexas.edu> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the debug patch. When I apply it, g++ complains
>>>>>>>>>>>> about "core/elf.cc:36:118: error: expected ‘)’ before ‘__VA_OPT__’"
>>>>>>>>>>>> I think __VA_OPT__ is only added to c++2a according to
>>>>>>>>>>>> https://gcc.gnu.org/onlinedocs/cpp/Variadic-Macros.html. Gcc
>>>>>>>>>>>> 7.4.0 on Ubuntu cannot compile that.
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Dec 10, 2019 at 7:20 AM Waldek Kozaczuk <
>>>>>>>>>>>> jwkoz...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> You may want to try to apply this patch -
>>>>>>>>>>>>> https://groups.google.com/forum/#!topic/osv-dev/LbnnY2Kcmak -
>>>>>>>>>>>>> that should provide many useful debug printouts.
>>>>>>>>>>>>>
>>>>>>>>>>>>> There is another patch I have sent that fixes the versioned
>>>>>>>>>>>>> self-lookup problem -
>>>>>>>>>>>>> https://groups.google.com/forum/#!topic/osv-dev/d56plMGXi6E.
>>>>>>>>>>>>> I wonder if it fixes your problem (my gut tells me yours is
>>>>>>>>>>>>> different).
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tuesday, December 10, 2019 at 3:07:09 AM UTC-5, Nadav
>>>>>>>>>>>>> Har'El wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Dec 9, 2019 at 10:51 PM zhiting zhu <zzt...@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm encountering this when I'm using some tensorflow
>>>>>>>>>>>>>>> functions:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> /lib/python3.6/tensorflow/python/_pywrap_tensorflow_internal.so:
>>>>>>>>>>>>>>> failed looking up symbol
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is interesting, because the "failed looking up symbol"
>>>>>>>>>>>>>> message is always followed by the name of the symbol looked up:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> core/elf.cc: abort("%s: failed looking up symbol
>>>>>>>>>>>>>> %s\n", pathname().c_str(), demangle(name).c_str());
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> You can try to add printouts in object::arch_relocate_rela()
>>>>>>>>>>>>>> to try to understand which symbol() is being called
>>>>>>>>>>>>>> with an empty name.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [backtrace]
>>>>>>>>>>>>>>> 0x00000000403442a7 <elf::object::symbol(unsigned int,
>>>>>>>>>>>>>>> bool)+391>
>>>>>>>>>>>>>>> 0x0000000040397dce <elf::object::arch_relocate_rela(unsigned
>>>>>>>>>>>>>>> int, unsigned int, void*, long)+574>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 0x000000004033eed4 <elf::object::relocate_rela()+196>
>>>>>>>>>>>>>>> 0x0000000040341d27 <elf::object::relocate()+199>
>>>>>>>>>>>>>>> 0x0000000040345623
>>>>>>>>>>>>>>> <elf::program::load_object(std::__cxx11::basic_string<char,
>>>>>>>>>>>>>>> std::char_traits<char>, std::allocator<char> >,
>>>>>>>>>>>>>>> std::vector<std::__cxx11::basic_string<char,
>>>>>>>>>>>>>>> std::char_traits<char>,
>>>>>>>>>>>>>>> std::allocator<char> >,
>>>>>>>>>>>>>>> std::allocator<std::__cxx11::basic_string<char,
>>>>>>>>>>>>>>> std::char_traits<char>, std::allocator<char> > > >,
>>>>>>>>>>>>>>> std::vector<std::shared_ptr<elf::object>,
>>>>>>>>>>>>>>> std::allocator<std::shared_ptr<elf::object> > >&)+1459>
>>>>>>>>>>>>>>> 0x0000000040345e70
>>>>>>>>>>>>>>> <elf::program::get_library(std::__cxx11::basic_string<char,
>>>>>>>>>>>>>>> std::char_traits<char>, std::allocator<char> >,
>>>>>>>>>>>>>>> std::vector<std::__cxx11::basic_string<char,
>>>>>>>>>>>>>>> std::char_traits<char>,
>>>>>>>>>>>>>>> std::allocator<char> >,
>>>>>>>>>>>>>>> std::allocator<std::__cxx11::basic_string<char,
>>>>>>>>>>>>>>> std::char_traits<char>, std::allocator<char> > > >, bool)+336>
>>>>>>>>>>>>>>> 0x0000000040465fd8 <dlopen+136>
>>>>>>>>>>>>>>> 0x0000100000937228 <_PyImport_FindSharedFuncptr+376>
>>>>>>>>>>>>>>> 0x745f70617277796f <???+1920432495>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It seems the name is not print out.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If I'm using the check-libcfunc-avail.sh to check the
>>>>>>>>>>>>>>> _pywrap_tensorflow_internal.so, I get the following output:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> pthread_mutex_consistent not found
>>>>>>>>>>>>>>> pthread_mutexattr_setrobust not found
>>>>>>>>>>>>>>> fmaf not found
>>>>>>>>>>>>>>> fma not found
>>>>>>>>>>>>>>> mallinfo not found
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> All of these we should eventually add, these are real Linux
>>>>>>>>>>>>>> glibc functions...
>>>>>>>>>>>>>> Feel free to open issues about them, or propose patches -
>>>>>>>>>>>>>> some will be trivial (e.g., fma() is already defined in musl/ we
>>>>>>>>>>>>>> just need
>>>>>>>>>>>>>> to enable it in the Makefile).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 0000000000000000 not found
>>>>>>>>>>>>>>> DF not found
>>>>>>>>>>>>>>> *UND* not found
>>>>>>>>>>>>>>> 0000000000000000 not found
>>>>>>>>>>>>>>> (GLIBC_2.2.5) not found
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> These may be (hopefully) parsing errors by the script?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> mremap not found
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Eventually will need to do this too...
>>>>>>>>>>>>>> https://github.com/cloudius-systems/osv/issues/184
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Originally, it also contains some output of missing complex
>>>>>>>>>>>>>>> number related functions. I include those from musl. The list
>>>>>>>>>>>>>>> reduces to
>>>>>>>>>>>>>>> the above.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It would be great to know which symbol it's missing. I'm not
>>>>>>>>>>>>>>> sure whether it's looking for a symbol from libc or from other
>>>>>>>>>>>>>>> library.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>> Zhiting
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>>>> Google Groups "OSv Development" group.
>>>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails
>>>>>>>>>>>>>>> from it, send an email to osv...@googlegroups.com.
>>>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/d84f4991-69d5-45d8-98a2-403257c9e4b2%40googlegroups.com
>>>>>>>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/d84f4991-69d5-45d8-98a2-403257c9e4b2%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>>>>>> .
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "OSv Development" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>>>> it, send an email to osv...@googlegroups.com.
>>>>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/4a72dd0b-b20a-4877-94da-aa9313add35e%40googlegroups.com
>>>>>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/4a72dd0b-b20a-4877-94da-aa9313add35e%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>>>>> .
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>> Google Groups "OSv Development" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to osv...@googlegroups.com.
>>>>>>>>>> To view this discussion on the web visit
>>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/a83feefd-2c93-48d0-bb9b-674ad0870853%40googlegroups.com
>>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/a83feefd-2c93-48d0-bb9b-674ad0870853%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "OSv Development" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to osv...@googlegroups.com.
>>>>>>>
>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/osv-dev/42454aa1-51b9-4638-bcd0-337002364524%40googlegroups.com
>>>>>>> <https://groups.google.com/d/msgid/osv-dev/42454aa1-51b9-4638-bcd0-337002364524%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>>> .
>>>>>>>
>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "OSv Development" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to osv...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/osv-dev/648f49e7-6f5a-4340-87ba-c923092c5af6%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/osv-dev/648f49e7-6f5a-4340-87ba-c923092c5af6%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "OSv Development" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to osv...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/osv-dev/CAL9cFfOGj_Jy1VagTiwNXew%3Dd7pyFdbDs7JbvpW88-hqKs27tw%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/osv-dev/CAL9cFfOGj_Jy1VagTiwNXew%3Dd7pyFdbDs7JbvpW88-hqKs27tw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "OSv Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to osv-dev+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/osv-dev/e82eb5f7-b3ee-4190-9ee6-7b85df302204%40googlegroups.com
>> <https://groups.google.com/d/msgid/osv-dev/e82eb5f7-b3ee-4190-9ee6-7b85df302204%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/CA%2B3q14wcCrCOFaPNZw-0qbpbc2cb30jdovDbTXUXNMRB8%2B_Q-Q%40mail.gmail.com.
