Re: [go-nuts] few questions about runtime implementation

[Hiroshi Ioka]

Honestly, I cannot imagine practical application of it. But I understand that 
Go tries to cover the case. 


Thank you for your answers.


Hiroshi


Sent with Unibox



> On Jul 20, 2017, at 2:40 AM, Ian Lance Taylor  wrote:
> 
> 
> On Wed, Jul 19, 2017 at 2:10 AM, Hiroshi Ioka  wrote:
>> 
>> 
>> 
>>> 
>>> 
>>> Before that change, newosproc0 was used when code built with
>>> 
>> 
>> -buildmode=c-archive did not use cgo.
>> 
>> 
>> Is there any use case for shared libraries that don’t have exported
>> functions?
>> 
> 
> 
> 
> 
> Go code can use init functions to run code that does operations
> without ever being directly called by non-Go code. There is a test of
> this in misc/cgo/testcshared/main2.c which runs the shared library in
> misc/cgo/testcshared/src/libgo2.
> 
> 
> Ian
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[Ian Lance Taylor]

On Wed, Jul 19, 2017 at 2:10 AM, Hiroshi Ioka  wrote:
>
>> Before that change, newosproc0 was used when code built with
> -buildmode=c-archive did not use cgo.
>
> Is there any use case for shared libraries that don’t have exported
> functions?

Go code can use init functions to run code that does operations
without ever being directly called by non-Go code.  There is a test of
this in misc/cgo/testcshared/main2.c which runs the shared library in
misc/cgo/testcshared/src/libgo2.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[Hiroshi Ioka]

Oh, I made a mistake. I mean shared => static/shared.




Sent with Unibox



> On Jul 19, 2017, at 6:10 PM, Hiroshi Ioka  wrote:
> 
> 
> Thank you, I understand the situation mostly.
> 
> 
>> Before that change, newosproc0 was used when code built with
> -buildmode=c-archive did not use cgo.
> 
> Is there any use case for shared libraries that don’t have exported functions?
> 
> 
> Hiroshi 
> 
> Sent with Unibox
> 
> 
> 
>> On Jul 19, 2017, at 7:40 AM, Ian Lance Taylor  wrote:
>> 
>> 
>> On Mon, Jul 17, 2017 at 8:51 AM, 井岡裕史  wrote:
>>> 
>>> 
>>> 
>>> 
>>> 
>>> c-share and c-archive enforce external linking.
>>> https://github.com/golang/go/blob/504deee6088c2448fc3e91c94a1ba69ec92fb7ef/src/cmd/link/internal/ld/config.go#L191-L196
>>> 
>>> 
>>> I think that external linking always use runtime/cgo because of following
>>> line.
>>> https://github.com/golang/go/blob/504deee6088c2448fc3e91c94a1ba69ec92fb7ef/src/cmd/link/internal/ld/lib.go#L459-L464
>>> 
>>> 
>>> So, I still cannot find any possibility that newosproc0 will be called.
>>> 
>> 
>> 
>> 
>> 
>> I think that is a bug in https://golang.org/cl/28971. The c-archive
>> build mode does not actually require external linking. The c-archive
>> build mode does not invoke the linker at all, so the choice between
>> internal and external linking is somewhat irrelevant.
>> 
>> 
>> Before that change, newosproc0 was used when code built with
>> -buildmode=c-archive did not use cgo.
>> 
>> 
>> Ian
>> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[Hiroshi Ioka]

Thank you, I understand the situation mostly.


> Before that change, newosproc0 was used when code built with
-buildmode=c-archive did not use cgo.

Is there any use case for shared libraries that don’t have exported functions?


Hiroshi 

Sent with Unibox



> On Jul 19, 2017, at 7:40 AM, Ian Lance Taylor  wrote:
> 
> 
> On Mon, Jul 17, 2017 at 8:51 AM, 井岡裕史  wrote:
>> 
>> 
>> 
>> 
>> 
>> c-share and c-archive enforce external linking.
>> https://github.com/golang/go/blob/504deee6088c2448fc3e91c94a1ba69ec92fb7ef/src/cmd/link/internal/ld/config.go#L191-L196
>> 
>> 
>> I think that external linking always use runtime/cgo because of following
>> line.
>> https://github.com/golang/go/blob/504deee6088c2448fc3e91c94a1ba69ec92fb7ef/src/cmd/link/internal/ld/lib.go#L459-L464
>> 
>> 
>> So, I still cannot find any possibility that newosproc0 will be called.
>> 
> 
> 
> 
> 
> I think that is a bug in https://golang.org/cl/28971. The c-archive
> build mode does not actually require external linking. The c-archive
> build mode does not invoke the linker at all, so the choice between
> internal and external linking is somewhat irrelevant.
> 
> 
> Before that change, newosproc0 was used when code built with
> -buildmode=c-archive did not use cgo.
> 
> 
> Ian
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[Ian Lance Taylor]

On Mon, Jul 17, 2017 at 8:51 AM, 井岡裕史  wrote:
>
> c-share and c-archive enforce external linking.
> https://github.com/golang/go/blob/504deee6088c2448fc3e91c94a1ba69ec92fb7ef/src/cmd/link/internal/ld/config.go#L191-L196
>
> I think that external linking always use runtime/cgo because of following
> line.
> https://github.com/golang/go/blob/504deee6088c2448fc3e91c94a1ba69ec92fb7ef/src/cmd/link/internal/ld/lib.go#L459-L464
>
> So, I still cannot find any possibility that newosproc0 will be called.

I think that is a bug in https://golang.org/cl/28971.  The c-archive
build mode does not actually require external linking.  The c-archive
build mode does not invoke the linker at all, so the choice between
internal and external linking is somewhat irrelevant.

Before that change, newosproc0 was used when code built with
-buildmode=c-archive did not use cgo.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[Hiroshi Ioka]

I see. Thanks.




Sent with Unibox



> On Jul 18, 2017, at 1:03 AM, Aram Hăvărneanu  wrote:
> 
> 
>> nitpick. os3_solaris.go.go set stack size by itself, instead of
>> using default one. is it intentional?
>> 
> 
> 
> 
> 
> You mean why does it explicitely ask for a 2MB stack when 2MB stacks
> are the default? I can't remember the exact reason I did that but
> it was a workaround for some sort of bug.
> 
> 
> -- 
> Aram Hăvărneanu
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[Aram Hăvărneanu]

> nitpick. os3_solaris.go.go set stack size by itself, instead of
> using default one. is it intentional?

You mean why does it explicitely ask for a 2MB stack when 2MB stacks
are the default? I can't remember the exact reason I did that but
it was a workaround for some sort of bug.

-- 
Aram Hăvărneanu

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[井岡裕史]

s/sys_solaris_amd64.go/os3_solaris.go/

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[井岡裕史]

Thank you for your detailed answer.

1. I'm convinced. I understand that I don't have to consider the difference
in general.

> On Solaris there is no difference because they create threads the
same way in all cases.

nitpick. sys_solaris_amd64.go set stack size by itself, instead of using
default one. is it intentional?

2.

> You are correct that on ELF platforms using c-shared or c-archive
> seems to force external linking, but just as I explained earlier
> about cgo, the build mode and the link mode are more or less
> conceptually orthogonal. That being said I do not know if c-shared
> and c-archive can be made to work with internal linking or not, nor
> what are the specific implications of that configuration.

c-share and c-archive enforce external linking.
https://github.com/golang/go/blob/504deee6088c2448fc3e91c94a
1ba69ec92fb7ef/src/cmd/link/internal/ld/config.go#L191-L196

I think that external linking always use runtime/cgo because of following
line.
https://github.com/golang/go/blob/504deee6088c2448fc3e91c94a
1ba69ec92fb7ef/src/cmd/link/internal/ld/lib.go#L459-L464

So, I still cannot find any possibility that newosproc0 will be called.

Hiroshi

On 17 July 2017 at 22:46, Aram Hăvărneanu  wrote:

> > 1. Is there any noticeable benchmark difference between internal
> > linking and external linking?
>
> A benchmark testing what?
>
> > IIUC, when I use external linking, runtime/cgo is used for OS thread
> > creation. In that case Go doesn't allocate the system stack itself.
>
> It's more complicated than that. By default, user-usage of cgo (when
> *you* import "C") uses external linking, stdlib-usage of cgo (e.g.
> os/user) uses internal linking, and non-cgo builds tend to use
> internal linking, however the type of linking (internal or external)
> is orthogonal to whether cgo (so runtime/cgo) is used or not.
>
> You can also use cgo with internal linking (except on systems which
> don't support it), and you can use cgo with internal linking (on
> system that support it).
>
> So your question is really about cgo, and not about linking.
>
> That being said, using runtime/cgo will indeed change the thread
> creation method on some systems, but not on others.
>
> On Solaris threads are always created the same way irrespective
> whether cgo is used or not.
>
> Plan 9 doesn't have cgo at all, so the point is moot.
>
> On Windows threads are created very similarly in both cgo and non-cgo
> cases, but there is a slight difference.
>
> On these three platforms the kernel always provides the system stack
> even when internal linking is used.
>
> On other systems like Linux and FreeBSD there indeed are fundemental
> differences to how threads are created and how stacks are allocated
> in the cgo and non-cgo cases. As you've surely discovered, on these
> other systems (most systems, really) Go allocates the system stacks
> before creating the threads, which are instructed to use this
> already-created stack.
>
> > So, I assume there are some difference related to memory usage.
>
> On Solaris there is no difference because they create threads the
> same way in all cases.
>
> On Windows, when cgo is used, the default thread stack size of 2MB
> is used for the system stack. When cgo is not used the threads are
> created with a 128kB stack instead.
>
> On other systems like Linux and FreeBSD some multiple of 8kB (usually
> 1) is used for the system stacks in the non-cgo case. In the cgo
> case, the default thread stack size is used (probably 2MB). I am
> ignoring non-standard build modes like c-shared for this discussion.
> The exact details vary in those cases but its the same fundamental
> idea.
>
> However one must note that this is virtual memory, not physical
> memory. These stacks only use address space, they only use as much
> physical memory as needed, usually one or two pages of memory (a
> page is usually 4kB but different from system to system), possibly
> with the exception of Windows which might require the use of use
> more physical memory.
>
> In principle, if you explicitely use cgo (make actual use of it)
> you might cause more physical memory to be mapped onto your stacks,
> than if you don't explicitely use cgo, but if you compare a pure
> Go program that uses runtime/cgo, with the same one that doesn't
> (perhaps due to changing linking method), there shouldn't be any
> difference in physical memory usage to a first approximation
>
> > Is runtime.newosproc0 reachable?  IIUC, newosproc0 is called by
> > _rt0_$GOARCH_$GOOS_lib if runtime/cgo is not used.  However,
> > _rt0_$GOARCH_$GOOS_lib is invoked only if buildmode=c-shared or
> > buildmode=c-archive. In addition, c-shared and c-archive always use
> > external linking. When is newosproc0 actually called?
>
> You are correct that on ELF platforms using c-shared or c-archive
> seems to force external linking, but just as I explained earlier
> about cgo, the build mode and the link mode are more or less
> conceptually 

Re: [go-nuts] few questions about runtime implementation

[Aram Hăvărneanu]

> 1. Is there any noticeable benchmark difference between internal
> linking and external linking?

A benchmark testing what?

> IIUC, when I use external linking, runtime/cgo is used for OS thread
> creation. In that case Go doesn't allocate the system stack itself.

It's more complicated than that. By default, user-usage of cgo (when
*you* import "C") uses external linking, stdlib-usage of cgo (e.g.
os/user) uses internal linking, and non-cgo builds tend to use
internal linking, however the type of linking (internal or external)
is orthogonal to whether cgo (so runtime/cgo) is used or not.

You can also use cgo with internal linking (except on systems which
don't support it), and you can use cgo with internal linking (on
system that support it).

So your question is really about cgo, and not about linking.

That being said, using runtime/cgo will indeed change the thread
creation method on some systems, but not on others.

On Solaris threads are always created the same way irrespective
whether cgo is used or not.

Plan 9 doesn't have cgo at all, so the point is moot.

On Windows threads are created very similarly in both cgo and non-cgo
cases, but there is a slight difference.

On these three platforms the kernel always provides the system stack
even when internal linking is used.

On other systems like Linux and FreeBSD there indeed are fundemental
differences to how threads are created and how stacks are allocated
in the cgo and non-cgo cases. As you've surely discovered, on these
other systems (most systems, really) Go allocates the system stacks
before creating the threads, which are instructed to use this
already-created stack.

> So, I assume there are some difference related to memory usage.

On Solaris there is no difference because they create threads the
same way in all cases.

On Windows, when cgo is used, the default thread stack size of 2MB
is used for the system stack. When cgo is not used the threads are
created with a 128kB stack instead.

On other systems like Linux and FreeBSD some multiple of 8kB (usually
1) is used for the system stacks in the non-cgo case. In the cgo
case, the default thread stack size is used (probably 2MB). I am
ignoring non-standard build modes like c-shared for this discussion.
The exact details vary in those cases but its the same fundamental
idea.

However one must note that this is virtual memory, not physical
memory. These stacks only use address space, they only use as much
physical memory as needed, usually one or two pages of memory (a
page is usually 4kB but different from system to system), possibly
with the exception of Windows which might require the use of use
more physical memory.

In principle, if you explicitely use cgo (make actual use of it)
you might cause more physical memory to be mapped onto your stacks,
than if you don't explicitely use cgo, but if you compare a pure
Go program that uses runtime/cgo, with the same one that doesn't
(perhaps due to changing linking method), there shouldn't be any
difference in physical memory usage to a first approximation

> Is runtime.newosproc0 reachable?  IIUC, newosproc0 is called by
> _rt0_$GOARCH_$GOOS_lib if runtime/cgo is not used.  However,
> _rt0_$GOARCH_$GOOS_lib is invoked only if buildmode=c-shared or
> buildmode=c-archive. In addition, c-shared and c-archive always use
> external linking. When is newosproc0 actually called?

You are correct that on ELF platforms using c-shared or c-archive
seems to force external linking, but just as I explained earlier
about cgo, the build mode and the link mode are more or less
conceptually orthogonal. That being said I do not know if c-shared
and c-archive can be made to work with internal linking or not, nor
what are the specific implications of that configuration.

On Windows c-shared and c-archive don't seem to force external
linking, but I am not sure whether internal linking is supposed to
work or not.

-- 
Aram Hăvărneanu

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] few questions about runtime implementation

[hirochachacha]

Hi, I have few questions out of curiosity.

1. Is there any noticeable benchmark difference between internal linking 
and external linking?
  IIUC, when I use external linking, runtime/cgo is used for OS thread 
creation. In that case Go doesn't allocate the system stack itself.
  So, I assume there are some difference related to memory usage. Can I see 
some benchmark result if any?

2. Is runtime.newosproc0 reachable?
  IIUC, newosproc0 is called by _rt0_$GOARCH_$GOOS_lib if runtime/cgo is 
not used.
  However, _rt0_$GOARCH_$GOOS_lib is invoked only if buildmode=c-shared or 
buildmode=c-archive.
  In addition, c-shared and c-archive always use external linking. When is 
newosproc0 actually called?

Thanks,

Hiroshi

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.