Re: [go-nuts] cgo Handle doc
On Friday, November 25, 2022 at 6:12:01 PM UTC+1 Ian Lance Taylor wrote:
> On Fri, Nov 25, 2022 at 12:06 AM Frédéric De Jaeger
> >
> > Thanks for your reply. I was really missing an essential piece.
> > I found a bit weird that you need to write a helper to pass a uintptr to
> a void* C function. It could be nice to allow that
> >
> > func Foo(x uintptr) {
> > C.foo((*C.char)x)
> > }
> >
> > but unfortunately, the compiler does not seem to accept it. Is there a
> reason ?
>
> In Go you can't convert a uintptr to an arbitrary pointer. You could
> write C.foo((*C.char)(unsafe.Pointer(x))). The compiler would accept
> that, but it would break the unsafe.Pointer conversion rules.
>
I was not asking for a conversion of a uintptr to an arbitrary pointer,
that would way too unsafe; but a conversion of uintptr to an arbitrary C
pointer (the same kind of typing rule that currently exists to allow casts
back and forth between uintptr and unsafe.Pointer). It would have the
obvious semantic, with no interference with the runtime.
Seeing other issues like
https://groups.google.com/g/golang-nuts/c/wcG1vKnDVkc, that would help in
many places. There is no compatibility issue. I don't think it makes cgo
any more dangerous to use. For that to be fully useful, the compiler would
have to accept the type *C.void in a cast expression.
Alternatively, you could provide a type `cgo.Pointer`, that has the same
typing rule as unsafe.Pointer, but has no pointer semantic. But that would
be too confusing, I don't like it.
>
> > From your last paragraph, can I do that ?
> >
> > /*
> > #include
> > static void* IntToPtr(uintptr_t v) { return (void*)v; }
> > */
> > import "C"
> >
> > import (
> > "runtime/cgo"
> > "unsafe"
> > )
> >
> > func HandleAsPointer(h cgo.Handle) unsafe.Pointer {
> > return C.IntToPtr(C.uintptr_t(uintptr(h)))
> > }
> >
> > Here the cast int -> void* happens in the C world and the conversion
> back to unsafe.Pointer looks legit according to the rule of cgo. But this
> will create an unsafe.Pointer containing a totally invalid pointer. Will it
> work (practically and theoretically) ? Is the GC robust against the
> presence of a random value in an unsafe.Pointer ?
>
> This won't work. As you say, the GC can't handle an invalid pointer
> value that is marked as a pointer.
>
ho. Then I think that might be good to document in cgo, close to this
sentence: *"The C type void* is represented by Go's unsafe.Pointer."*
that it is illegal to put an invalid pointer in an unsafe.Pointer.
thanks for your answers, this is very enlightening
Fred
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/e992a773-ffe0-4b82-b46d-6184fb66f9dan%40googlegroups.com.
Re: [go-nuts] cgo Handle doc
Given the *special cases* there
https://pkg.go.dev/cmd/cgo#hdr-Special_cases, I would say this is not a
good idea.
On Friday, November 25, 2022 at 9:06:05 AM UTC+1 Frédéric De Jaeger wrote:
> Thanks for your reply. I was really missing an essential piece.
> I found a bit weird that you need to write a helper to pass a uintptr to a
> void* C function. It could be nice to allow that
>
> func Foo(x uintptr) {
> C.foo((*C.char)x)
> }
>
> but unfortunately, the compiler does not seem to accept it. Is there a
> reason ?
>
> Anyway, I suppose the issue is that the runtime will interpret the
> uintptr, when cast to a unsafe.Pointer as a potential valid pointer and if
> it looks a bit too much like a legit go pointer, that might confuse the
> GC. I suppose that unsafe.Pointer can safely handle C pointers (from the
> doc of cgo)
>
> If the uintptr contains a valid C pointer, do we have a practical issue,
> like with the following code ?
>
> func main() {
>v := uintptr(unsafe.Pointer(C.CString("Hello from stdio")))
>C.myprint(unsafe.Pointer(v))
>C.free(unsafe.Pointer(cs))
> }
>
> This is a question about the current implementation, I know it violates
> the rule of unsafe.Pointer. This is just to have a better understanding
> about what the runtime does and when (at conversion time? C call time ? GC
> time ?)
>
>
> From your last paragraph, can I do that ?
>
> /*
> #include
> static void* IntToPtr(uintptr_t v) { return (void*)v; }
> */
> import "C"
>
> import (
>"runtime/cgo"
>"unsafe"
> )
>
> func HandleAsPointer(h cgo.Handle) unsafe.Pointer {
>return C.IntToPtr(C.uintptr_t(uintptr(h)))
> }
>
> Here the cast int -> void* happens in the C world and the conversion back
> to unsafe.Pointer looks legit according to the rule of cgo. But this will
> create an unsafe.Pointer containing a totally invalid pointer. Will it
> work (practically and theoretically) ? Is the GC robust against the
> presence of a random value in an unsafe.Pointer ?
>
> Thanks
>
>
> On Wednesday, November 23, 2022 at 9:31:45 PM UTC+1 Ian Lance Taylor wrote:
>
>> On Wed, Nov 23, 2022 at 9:25 AM Frédéric De Jaeger
>> wrote:
>> >
>> > There is something that puzzles me a lot in the doc about cgo.Handle
>> there https://pkg.go.dev/runtime/cgo. It says:
>> >
>> > Some C functions accept a void* argument that points to an arbitrary
>> data value supplied by the caller. It is not safe to coerce a cgo.Handle
>> (an integer) to a Go unsafe.Pointer, but instead we can pass the address of
>> the cgo.Handle to the void* parameter, as in this variant of the previous
>> example:
>> >
>> > I was under the impression that casting a uintptr to an unsafe.Pointer
>> just to call a C function (that accepts a void*) is somewhat valid. (This
>> is not clearly specified as a valid case in the doc of unsafe.Pointer, so,
>> it might be invalid from a very pedantic point of view).
>> >
>> > The advice given in the doc looks worst. Taking the address of a local
>> variable and giving it to C looks even more undefined (if C stores the
>> pointer and reuse it after the original call returns).
>> >
>> > Don't we have a doc issue there ?
>> >
>> > I know that in C, the conversion uintptr -> void* is subject to debate
>> (trap representation, ...). In practice, nobody cares because It just
>> works. (example of the nobody cares
>> https://www.freedesktop.org/software/gstreamer-sdk/data/docs/latest/glib/glib-Type-Conversion-Macros.html)
>>
>>
>> >
>> > In practice, I'm almost certain that the naïve code that just gives the
>> cgo.Handle to a C function as a void* is *much* safer that the pattern
>> suggested in the doc.
>> >
>> > So should the doc be changed, or am I missing some important piece ?
>>
>> It is not safe to convert a uintptr to an unsafe.Pointer when calling
>> a C function. The docs in the unsafe package are intended to be
>> precise, and in particular the compiler is aware of them. The
>> compiler applies special handling to a conversion from unsafe.Pointer
>> to uintptr in the argument to a function with no body like
>> syscall.Syscall. That special handling does not occur in calls to cgo
>> functions. While it will often work in practice, there are cases
>> where it will fail.
>>
>> One way to handle this kind of case is to write a tiny C wrapper in
>> the cgo comment that takes a uintptr and does the conversion to void*
>> in C. That conversion is safe, because the Go runtime will not be
>> aware of it and will never mistakenly treat the uintptr value as a
>> pointer value.
>>
>> Ian
>>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/b42d57a0-c1bd-43ad-8679-4a376b211e81n%40googlegroups.com.
Re: [go-nuts] cgo Handle doc
Thanks for your reply. I was really missing an essential piece.
I found a bit weird that you need to write a helper to pass a uintptr to a
void* C function. It could be nice to allow that
func Foo(x uintptr) {
C.foo((*C.char)x)
}
but unfortunately, the compiler does not seem to accept it. Is there a
reason ?
Anyway, I suppose the issue is that the runtime will interpret the uintptr,
when cast to a unsafe.Pointer as a potential valid pointer and if it looks
a bit too much like a legit go pointer, that might confuse the GC. I
suppose that unsafe.Pointer can safely handle C pointers (from the doc of
cgo)
If the uintptr contains a valid C pointer, do we have a practical issue,
like with the following code ?
func main() {
v := uintptr(unsafe.Pointer(C.CString("Hello from stdio")))
C.myprint(unsafe.Pointer(v))
C.free(unsafe.Pointer(cs))
}
This is a question about the current implementation, I know it violates the
rule of unsafe.Pointer. This is just to have a better understanding about
what the runtime does and when (at conversion time? C call time ? GC time
?)
>From your last paragraph, can I do that ?
/*
#include
static void* IntToPtr(uintptr_t v) { return (void*)v; }
*/
import "C"
import (
"runtime/cgo"
"unsafe"
)
func HandleAsPointer(h cgo.Handle) unsafe.Pointer {
return C.IntToPtr(C.uintptr_t(uintptr(h)))
}
Here the cast int -> void* happens in the C world and the conversion back
to unsafe.Pointer looks legit according to the rule of cgo. But this will
create an unsafe.Pointer containing a totally invalid pointer. Will it
work (practically and theoretically) ? Is the GC robust against the
presence of a random value in an unsafe.Pointer ?
Thanks
On Wednesday, November 23, 2022 at 9:31:45 PM UTC+1 Ian Lance Taylor wrote:
> On Wed, Nov 23, 2022 at 9:25 AM Frédéric De Jaeger
> wrote:
> >
> > There is something that puzzles me a lot in the doc about cgo.Handle
> there https://pkg.go.dev/runtime/cgo. It says:
> >
> > Some C functions accept a void* argument that points to an arbitrary
> data value supplied by the caller. It is not safe to coerce a cgo.Handle
> (an integer) to a Go unsafe.Pointer, but instead we can pass the address of
> the cgo.Handle to the void* parameter, as in this variant of the previous
> example:
> >
> > I was under the impression that casting a uintptr to an unsafe.Pointer
> just to call a C function (that accepts a void*) is somewhat valid. (This
> is not clearly specified as a valid case in the doc of unsafe.Pointer, so,
> it might be invalid from a very pedantic point of view).
> >
> > The advice given in the doc looks worst. Taking the address of a local
> variable and giving it to C looks even more undefined (if C stores the
> pointer and reuse it after the original call returns).
> >
> > Don't we have a doc issue there ?
> >
> > I know that in C, the conversion uintptr -> void* is subject to debate
> (trap representation, ...). In practice, nobody cares because It just
> works. (example of the nobody cares
> https://www.freedesktop.org/software/gstreamer-sdk/data/docs/latest/glib/glib-Type-Conversion-Macros.html
> )
> >
> > In practice, I'm almost certain that the naïve code that just gives the
> cgo.Handle to a C function as a void* is *much* safer that the pattern
> suggested in the doc.
> >
> > So should the doc be changed, or am I missing some important piece ?
>
> It is not safe to convert a uintptr to an unsafe.Pointer when calling
> a C function. The docs in the unsafe package are intended to be
> precise, and in particular the compiler is aware of them. The
> compiler applies special handling to a conversion from unsafe.Pointer
> to uintptr in the argument to a function with no body like
> syscall.Syscall. That special handling does not occur in calls to cgo
> functions. While it will often work in practice, there are cases
> where it will fail.
>
> One way to handle this kind of case is to write a tiny C wrapper in
> the cgo comment that takes a uintptr and does the conversion to void*
> in C. That conversion is safe, because the Go runtime will not be
> aware of it and will never mistakenly treat the uintptr value as a
> pointer value.
>
> Ian
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/dc4eba12-cb08-4497-8d57-255a8213e608n%40googlegroups.com.
[go-nuts] cgo Handle doc
Hi, There is something that puzzles me a lot in the doc about cgo.Handle there https://pkg.go.dev/runtime/cgo. It says: *Some C functions accept a void* argument that points to an arbitrary data value supplied by the caller. It is not safe to coerce a cgo.Handle (an integer) to a Go unsafe.Pointer, but instead we can pass the address of the cgo.Handle to the void* parameter, as in this variant of the previous example: * I was under the impression that casting a uintptr to an unsafe.Pointer just to call a C function (that accepts a void*) is somewhat valid. (This is not clearly specified as a valid case in the doc of unsafe.Pointer, so, it might be invalid from a very pedantic point of view). The advice given in the doc looks worst. Taking the address of a local variable and giving it to C looks even more undefined (if C stores the pointer and reuse it after the original call returns). Don't we have a doc issue there ? I know that in C, the conversion uintptr -> void* is subject to debate (trap representation, ...). In practice, nobody cares because *It just works*. (example of the nobody cares https://www.freedesktop.org/software/gstreamer-sdk/data/docs/latest/glib/glib-Type-Conversion-Macros.html) In practice, I'm almost certain that the naïve code that just gives the cgo.Handle to a C function as a void* is *much* safer that the pattern suggested in the doc. So should the doc be changed, or am I missing some important piece ? Thanks Fred -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/49cadd8f-89b4-4f91-8ad3-241768758dc3n%40googlegroups.com.
Re: [go-nuts] about -buildmode c-shared
On Saturday, February 13, 2021 at 4:21:34 PM UTC+1 Ian Lance Taylor wrote: > > > > I was naively assuming that the unloading issue is easy to tackle when > the runtime is not shared between .so. Is it true ? > > No. The current Go runtime has no ability to shut down all > goroutines. If any goroutines are left running, and the plugin is > unloaded, the program will crash. > > I was not assuming a synchronous uncooperative way for the runtime to kill a running goroutine. I was more thinking about some cooperative strategy. Like a running go routine could check sometimes (maybe in the function prologue, or when doing an allocation) that it is time to die. Anycase, that could be also a constraint of the business code in the plugin. Like in C, they are not supposed to be actively running when the .so is *dlclosed*. In fact, what happens in practice today if we *dlclose* a plugin that has no running goroutine (and also no cgo invocations). Does it "*work*" somehow ? Are we leaking threads, waiting on some dead semaphore or something ? > > A last technical question, do you link the shared object with the ld > flag `- z nodelete` (which turn dlclose into no op). > > As far as I know we do not. > > This `ld` flag makes me think of another approach. Maybe it is ok to just turn off the effective unloading of the .so (with that ld flag). So *dlclose* is no-op. But if the business code takes care of cleaning most of its global state, maybe the gc has the opportunity to return some memory back to the OS (well, since the gc is not moving stuff, maybe the hope is slim, I don't know). That approach does not solve the problem of the developer iteratively working on a plugin. Apart from that, this looks like an acceptable compromise for someone who wants to ship an loadable plugin that can be " *unloaded*". At least it *works* somehow, leaking a bit of memory. But it does not crash. Of course, it is expected that if I load *libfoo.so* unload it, and load it again, everything works. I don't see why it would not. The second *dlopen* is presumably no-op. Now about the unique runtime constraint. In the previous message, you said that we might be able to run several runtimes in the same process (private runtime symbols, ...) If that works, it would mean that if the runtime symbols are public, but *versioned*, then maybe the runtime constraint is gone, isn't it? If two lib *libfoo.so* and *libbar.so* are built with different runtimes, we get two instantiated runtime in the binary that loads them simultaneously. Well, too bad, we lose a bit of memory, but at least it works. It looks acceptable to me. In the favorable case, they share the same runtime and every one is happy. I am curious about what happens on MacOS today. Because my understanding of their *two level namespace *(https://developer.apple.com/library/archive/documentation/Porting/Conceptual/PortingUnix/compiling/compiling.html#//apple_ref/doc/uid/TP40002850-BCIHJBBF) implies that you instantiate a new runtime whenever you load a c-shared .dylib. I'm not sure at all. It seems there is a way to force a flat namespace (it does not seem to be the default, thought). I haven't verified that yet. By the way, what the most efficient way to detect if two runtime are being instantiated ? Something like a debug log that is supposed to happen once, or a similar trick. Thanks Fred -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/edc90dd0-9f6d-4fbb-8629-db52b11ede42n%40googlegroups.com.
Re: [go-nuts] about -buildmode c-shared
On Tuesday, February 9, 2021 at 2:46:33 AM UTC+1 Ian Lance Taylor wrote: > > > Thanks, you may be correct: it may be possible to hide all symbols > other than the ones that are intended to be exported to the C program. > If it is possible to do that reliably in all cases, then the effect > should be that if you link against multiple shared libraries, each > will have an independent Go runtime. The program will wind up with > different Go heaps and different garbage collectors competing with > each other. The result will be inefficient, sometimes alarmingly so, > but I don't immediately see any reason why it wouldn't work. > > No bad surprise expected from other kind of uniqu per process resources ? (signal? ...) I suppose you mean other kind of inefficiency than the basic ram usage ? I've observed that when I run several CPU intensive go apps on the same host, the cumulative GC pause (Memstats.PauseTotalNs) raises dramatically (not linear in the number of go process running). But in more gentle scenario, where the go plugin rarely runs, do we hit those inefficiencies ? At the moment, go can't be used reliably as a tool to write C plugin for a generic C host. Because of the runtime constraints, and the unloading issue. I was naively assuming that the unloading issue is easy to tackle when the runtime is not shared between .so. Is it true ? I can see a difficulty when there are running go routines doing cgo invocations (and we can probably fail/abort/crash in this case) . Apart from that, I have the feeling this should be easy to cleanup all the runtime ressources (memory, threads). I suppose this kind of cleanup logic was never written because exit(0) does the same job. IMHO, being able to write generic C plugin in go, would be a good selling feature (I would appreciate it as much as all the people here https://github.com/golang/go/issues/11100). So, suppose we implement that feature by hiding/unsharing the runtime (assuming it works and we fill in the missing pieces) The question is how bad would be that solution compared to the absence of that feature ? Are there really some business usage of the current existing feature ? (people really loading several `c-shared` go .so and expecting to share the runtime) What about yet another build mode like `c-shared-private-runtime` ? unloading would be only implemented with this build mode. A last technical question, do you link the shared object with the ld flag `- z nodelete*` *(which turn dlclose into no op). > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f1332fdc-67c4-492e-9810-02cfc95eda7bn%40googlegroups.com.
[go-nuts] about -buildmode c-shared
Hi, this is a followup of
https://groups.google.com/g/golang-dev/c/J7W3sDexK18
that I've mistakenly posted in the wrong channel. I feel ashamed about it.
Ian Lance Taylor wrotes:
>
*This is a consequence of the fact that (on most systems) shared libraries
share a symbol namespace, so that all references to, say, runtime.chansend
are resolved to the same function. If different Go shared libraries built
using -buildmode=c-shared are built with different versions of Go, then the
fact that they call the same runtime.chansend function can mean that one of
the shared libraries will not work correctly. *
Thanks for your quick response.
When I build a package like this
go build -work -buildmode c-shared -o pouet.so .
I get a standalone .so that contains anything needed (business code + go
runtime)
The contract is that symbols like this:
//export XPluginStart
func XPluginStart(outName, outSig, outDesc *C.char) C.int {
are exported and made visible to the outside world (the C based host
programm that loads the .so)
When I do:
> nm pouet.so | grep " T "
It gives many publicly visible go symbols
000d75a0 T x_cgo_callers
000d71d0 T x_cgo_init
000d7380 T x_cgo_mmap
000d73b0 T x_cgo_munmap
and also mine:
000d6be0 T XPluginStart
My understanding of that is that you can control the visibility of symbols,
can't you ?
IMHO, all the other symbols (including everything from the runtime) could
be hidden.
>
*That's not how shared libraries work on many systems. *
I suppose I might be missing something. I'm not a total expert on that
matter. Suppose we tune the visibility of symbols such that the previous
command would only shows the ones I specifically want to be accessible to
the C host. And everything else (runtime global variable, functions, ...)
is made private (it appears with the symbol `t` instead of `T`). Wouldn't
that guarantee that successive loading of plugin like these would all have
their own copy of the runtime ?
MacOS, linux, and windows all provide a way to control the symbols
visibility. Were you talking about other systems where you can't control
the symbol visibility so easily ?
MacOS has his _two level namespace_ thingy. It is supposed to guarantee
(if I understand that stuff correctly) that symbol resolution is done
*right*, (if `a.dylib` calls `foo` that exists in `a.dylib` and also in
another `b.dylib`, the one in `a.dylib` gets called) So what currently
happens on MacOS ? Don't we have several runtime instantiated in this
scenario ?
I can imagine other traps on that road. Other form of global per process
resources could cause issues. Signal, maybe ?
Fred
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/dc5f138d-9512-4ce4-9727-8b55c79fa3bbn%40googlegroups.com.
