Re: [PATCH] golang-example: Use shared and pie compilation flags

[Nadav Har'El]

On Wed, Aug 31, 2016 at 5:33 PM, Nadav Har'El  wrote:

> I'm more worried about the symbol(sym) part of this code... What if there
> is no symbol involved?  In this case (again see readelf --relocs for your
> object), there is no object...
>
> I think the code involved is the one added in commit
> b9b73661543e614c7dfb3616bd3a75acfa30911e:
>
> x64: Fix R_X86_64_TPOFF64 relocation without symbol
>
> Maybe this code (the no-symbol case) has a bug?
>
>
This actually eveloved quite a bit since that patch. It now looks like:

case R_X86_64_TPOFF64:
if (sym) {
auto sm = symbol(sym);
sm.obj->alloc_static_tls();
auto tls_offset = sm.obj->static_tls_end() +
sched::kernel_tls_size();
*static_cast(addr) = sm.symbol->st_value + addend -
tls_offset;
} else {
alloc_static_tls();
auto tls_offset = static_tls_end() + sched::kernel_tls_size();
*static_cast(addr) = addend - tls_offset;
}


This was written in commit 8e99d1537dc753410cca0197528118ed370ad589
specificially about support initial-exec tls model.
But maybe has a bug?

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] golang-example: Use shared and pie compilation flags

[Nadav Har'El]

On Wed, Aug 31, 2016 at 5:13 PM, Benoît Canet 
wrote:

>case R_X86_64_TPOFF64:
>// FIXME: assumes TLS segment comes before DYNAMIC segment
>

I'm not sure this code even runs when loading a shared object. I think it
only runs when loading the kernel, and there is another piece of code with
R_X86_64_TPOFF64 which gets used when loading objects.


>
>
> This is not the case:
>
> See in the dump:
>

I don't understand what I see in that dump.
To understand this FIXME, please refer to commit f09e0059:

I think the FIXME refers to that we need t->tls.size() to have been
previously set. You can add a printout to see if hasn't been set.

I'm more worried about the symbol(sym) part of this code... What if there
is no symbol involved?  In this case (again see readelf --relocs for your
object), there is no object...

I think the code involved is the one added in commit
b9b73661543e614c7dfb3616bd3a75acfa30911e:

x64: Fix R_X86_64_TPOFF64 relocation without symbol

Maybe this code (the no-symbol case) has a bug?



>
> 000c8d90 :
>   c8d90:   48 8b 0d 29 72 38 00mov0x387229(%rip),%rcx#
> 44ffc0 <_DYNAMIC+0x200>
>   c8d97:   64 48 8b 09 mov%fs:(%rcx),%rcx
>   c8d9b:   48 3b 61 10 cmp0x10(%rcx),%rsp
>   c8d9f:   76 60   jbec8e01 
>
>*static_cast(addr) = t->dyn_tabs.lookup(sym)->st_value -
> t->tls.size;
>break;
>case R_X86_64_IRELATIVE:
>
>
> On Wed, Aug 31, 2016 at 4:07 PM, Benoît Canet  > wrote:
>
>> This is what I saw in the code managing the flags.
>>
>> By anyway I think you are right and shared is an "impasse".
>>
>> On Wed, Aug 31, 2016 at 4:06 PM, Nadav Har'El  wrote:
>>
>>>
>>> On Wed, Aug 31, 2016 at 4:26 PM, Benoît Canet <
>>> ben...@cloudius-systems.com> wrote:
>>>

 Buildmode shared alone crash.



>>> the flags are |= between them.

>>>
>>> This is not what I see in https://golang.org/src/cmd/go/build.go
>>>
>>> But it seems these two options indeed have a funny interaction between
>>> them - shared enables "dynlink" which gets remembered even when the build
>>> mode is switched to PIE.
>>>
>>>
>>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] golang-example: Use shared and pie compilation flags

[Benoît Canet]

   case R_X86_64_TPOFF64:
   // FIXME: assumes TLS segment comes before DYNAMIC segment


This is not the case:

See in the dump:

000c8d90 :
  c8d90:   48 8b 0d 29 72 38 00mov0x387229(%rip),%rcx#
44ffc0 <_DYNAMIC+0x200>
  c8d97:   64 48 8b 09 mov%fs:(%rcx),%rcx
  c8d9b:   48 3b 61 10 cmp0x10(%rcx),%rsp
  c8d9f:   76 60   jbec8e01 

   *static_cast(addr) = t->dyn_tabs.lookup(sym)->st_value -
t->tls.size;
   break;
   case R_X86_64_IRELATIVE:


On Wed, Aug 31, 2016 at 4:07 PM, Benoît Canet 
wrote:

> This is what I saw in the code managing the flags.
>
> By anyway I think you are right and shared is an "impasse".
>
> On Wed, Aug 31, 2016 at 4:06 PM, Nadav Har'El  wrote:
>
>>
>> On Wed, Aug 31, 2016 at 4:26 PM, Benoît Canet <
>> ben...@cloudius-systems.com> wrote:
>>
>>>
>>> Buildmode shared alone crash.
>>>
>>>
>>>
>> the flags are |= between them.
>>>
>>
>> This is not what I see in https://golang.org/src/cmd/go/build.go
>>
>> But it seems these two options indeed have a funny interaction between
>> them - shared enables "dynlink" which gets remembered even when the build
>> mode is switched to PIE.
>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] golang-example: Use shared and pie compilation flags

[Benoît Canet]

This is what I saw in the code managing the flags.

By anyway I think you are right and shared is an "impasse".

On Wed, Aug 31, 2016 at 4:06 PM, Nadav Har'El  wrote:

>
> On Wed, Aug 31, 2016 at 4:26 PM, Benoît Canet  > wrote:
>
>>
>> Buildmode shared alone crash.
>>
>>
>>
> the flags are |= between them.
>>
>
> This is not what I see in https://golang.org/src/cmd/go/build.go
>
> But it seems these two options indeed have a funny interaction between
> them - shared enables "dynlink" which gets remembered even when the build
> mode is switched to PIE.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] golang-example: Use shared and pie compilation flags

[Nadav Har'El]

On Wed, Aug 31, 2016 at 4:26 PM, Benoît Canet 
wrote:

>
> Buildmode shared alone crash.
>
>
>
the flags are |= between them.
>

This is not what I see in https://golang.org/src/cmd/go/build.go

But it seems these two options indeed have a funny interaction between them
- shared enables "dynlink" which gets remembered even when the build mode
is switched to PIE.

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] golang-example: Use shared and pie compilation flags

[Nadav Har'El]

On Wed, Aug 31, 2016 at 4:26 PM, Benoît Canet 
wrote:

>
>
> Go use it's own tls see: http://blog.altoros.com/golang-internals-part-5-
> runtime-bootstrap-process.html
>

Thanks. Interesting.

It is very sad, and surprising, that Go does this: If I understand
correctly, it only has two variables in TLS - "g" and "m". It could have
re-used the existing C TLS ABI (see https://www.akkadia.org/drepper/tls.pdf)
whose main purpose is to allow different parts of the program (different
shared objects, etc.) to add their own TLS variables without needing one
entity to "take over" the fs register as Go does. Or it could have put the
f-ing variable, "g" in a register

I think we have two ways to proceed now:

1. Find a way where Go uses the C TLS ABI so it can co-exist with a C
program. I believe this is exactly the "c-shared" build mode (but correct
me if I'm wrong)... You tried to use it, but TLS didn't work. I assume we
still have some unknown bug in our "initial exec" TLS mode which Go uses
(there is a R_X86_64_TPOFF64 relocation in the shared object) which makes
this work incorrectly.

2. Give up on c-shared and go back to shared (or pie) which cannot co-exist
with C code which uses TLS.
It looks like this is the direction you're going in now.
Let's think what is the implication of this:

This option uses arch_prctl(), as you noticed, which sets the FS_BASE MSR.
But OSv (and other shared objects written in C and might be loaded) need
the FS_BASE MSR to be set up in a particular way - to point to OSv's TCB of
this thread and have a particular structure which the dynamic linker built
considering the need of all the loaded shared objects and the ELF TLS ABI.

SO, what this means is that:
1. After the arch_prctl() call, the thread which ran it cannot run OSv
code! When we switch to OSv code - either by calling a function (e.g.,
malloc()) or a system call - we need to temporarily restore OSv's FS_BASE,
and return it back when going back to the Go code.
2. Moreover, when we context switch out of the thread, we need to save its
FS_BASE and restore it when returning to the thread. We don't have exactly
that at the moment - currently, thread::switch_to() changes the FS_BASE to
the thread's fixed "_tcb" area, but when entering a thread which previously
used arch_prctl() we need to restore that address - not _tcb! So we need to
save and restore fs_base on context switch rather than set it to a fixed
address.

I really think this option will be very difficult to achieve, and fixing
c-shared will be easier. But maybe I'm missing something important here.

Something I don't understand now is how, even in Linux, the Go code, with
its funky FS_BASE setup, can use glibc functions like malloc(). What if
those functions use TLS (I would like to think that malloc() in particular
does)? Or does Go have special glue code to restore glibc's fs_base before
calling a glibc function?

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [PATCH] golang-example: Use shared and pie compilation flags

[Benoît Canet]

Buildmode shared alone crash.

the flags are |= between them.

Go use it's own tls see:
http://blog.altoros.com/golang-internals-part-5-runtime-bootstrap-process.html

On Wed, Aug 31, 2016 at 3:22 PM, Nadav Har'El  wrote:

>
> On Wed, Aug 31, 2016 at 4:05 PM, Benoit Canet  com> wrote:
>
>> The c-shared is intented to build C libraries hence it
>> does not export main and it's initialization does not
>> set the fs register with arch_prctl() leading to a crash.
>>
>> Use -buildmode=shared -buildmode=pie to produce a library
>>
>
> Doesn't the second option override the first one??
>
>
>> which export main and is position independant while having
>> and init calling arch_prctl().
>>
>
> I need either you or Pekka (who wrote this patch?) to explain to me how it
> makes sense for arch_prctl() to override the fs base, despite the fact that
> OSv needs to to be set to something else for OSv's internal TLS mechanisms
> to work.
>
> This still doesn't make any sense to me :-(
>
> Can you please try buildmode=shared instead of buildmode=pie? This should
> build a shared library which can't just (I would hope) take over the fs
> base register on a whim
>
>
>>
>> With this patch and the arch_prctl() stuff I revived from
>> Pekka's previous go work the code now crash after resolving
>> __vdso_gettimeofday to null later when the scheduler is initialized.
>>
>> Signed-off-by: Benoît Canet 
>> ---
>>  golang-example/Makefile | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/golang-example/Makefile b/golang-example/Makefile
>> index 15cc093..bbfb015 100644
>> --- a/golang-example/Makefile
>> +++ b/golang-example/Makefile
>> @@ -3,7 +3,7 @@ module: hello
>> echo '/hello: $${MODULE_DIR}/hello' > usr.manifest
>>
>>  hello: hello.go
>> -   /usr/lib/golang/bin/go build -buildmode=c-shared hello.go
>> +   /usr/lib/golang/bin/go build -buildmode=shared -buildmode=pie
>> hello.go
>>
>>  clean:
>> rm -f hello usr.manifest
>> --
>> 2.7.4
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "OSv Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to osv-dev+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PATCH] golang-example: Use shared and pie compilation flags

[Benoit Canet]

The c-shared is intented to build C libraries hence it
does not export main and it's initialization does not
set the fs register with arch_prctl() leading to a crash.

Use -buildmode=shared -buildmode=pie to produce a library
which export main and is position independant while having
and init calling arch_prctl().

With this patch and the arch_prctl() stuff I revived from
Pekka's previous go work the code now crash after resolving
__vdso_gettimeofday to null later when the scheduler is initialized.

Signed-off-by: Benoît Canet 
---
 golang-example/Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/golang-example/Makefile b/golang-example/Makefile
index 15cc093..bbfb015 100644
--- a/golang-example/Makefile
+++ b/golang-example/Makefile
@@ -3,7 +3,7 @@ module: hello
echo '/hello: $${MODULE_DIR}/hello' > usr.manifest
 
 hello: hello.go
-   /usr/lib/golang/bin/go build -buildmode=c-shared hello.go
+   /usr/lib/golang/bin/go build -buildmode=shared -buildmode=pie hello.go
 
 clean:
rm -f hello usr.manifest
-- 
2.7.4

-- 
You received this message because you are subscribed to the Google Groups "OSv 
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to osv-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.