Re: [go-nuts] cgo: passing unsafe.Pointer of Slice without explicit pinning?
Unless, the C function stores the pointer and it's kept after the call returns, correct? That's what I mean by calling the function keep_this_pointer in the second example. In this case, it would be necessary, I'm assuming? Julio On Monday, July 8, 2024 at 11:56:30 PM UTC+2 Ian Lance Taylor wrote: > On Mon, Jul 8, 2024 at 12:38 PM Antonio Caceres Cabrera > wrote: > > > > Sorry, accidentally hit the wrong response button, so I'm posting it > again: > > > > Thanks for the clarification, Ian. > > > > Is it also possible to pin memory to local go-arrays? > > The docs state > > >Go values created by calling new, by taking the address of a composite > literal, or by taking the address of a local variable may also have their > memory pinned using runtime.Pinner. > > > > In this example: > > ``` > > var pin runtime.Pinner > > var buf [32]byte > > pin.Pin([0]) > > C.keep_this_pointer([0]) > > ``` > > > > Is this ok since it counts as taking the address of a local variable or > would the array have to be created with `new([32]byte)` ? > > Yes, this is an example of taking the address of a local variable. > The runtime.Pinner is not required here. > > In general a runtime.Pinner is only required when passing the address > of a value that itself contains Go pointers. In that case the > internal Go pointers need to be explicitly pinned. > > Ian > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/b3b3a36b-87ba-4271-b004-810670eb759an%40googlegroups.com.
Re: [go-nuts] cgo: passing unsafe.Pointer of Slice without explicit pinning?
Sorry, accidentally hit the wrong response button, so I'm posting it again:
Thanks for the clarification, Ian.
Is it also possible to pin memory to local go-arrays?
The docs state
>Go values created by calling new, by taking the address of a composite
literal, or by taking the address of a local variable may also have their
memory pinned using runtime.Pinner.
In this example:
```
var pin runtime.Pinner
var buf [32]byte
pin.Pin([0])
C.keep_this_pointer([0])
```
Is this ok since it counts as taking the address of a local variable or
would the array have to be created with `new([32]byte)` ?
Julio
On Sunday, July 7, 2024 at 5:38:13 PM UTC+2 Ian Lance Taylor wrote:
> On Sun, Jul 7, 2024 at 6:17 AM Antonio Caceres Cabrera
> wrote:
> >
> > I'm trying to use cgo for a C library, specifically a function which
> takes a void pointer to a buffer, and I'm trying to pass a Go-Slice as this
> buffer.
> >
> > ```
> > func Write(buf []byte) (int, error) {
> > // var pin rumtime.Pinner
> > // pin.Pin([0])
> > // defer pin.Unpin()
> > // is this necessary? Is it even safe?
> > ...
> > ret := C.lib_read(unsafe.Pointer([0]), C.int(len(buf))) // function
> which reads from this buffer
> > ...
> > }
> > ```
> > Does this require explicit pinning of buf? And if so, is the commented
> part a valid pinning of a slice? The documentation states that pointers
> passed to C functions are implicitly pinned (on the top-level that is), but
> does this also apply to slices, i.e. if I pass a (unsafe) pointer to the
> first element, is the slice's entire backing array implicitly pinned?
> >
> > I know that there are some runtime checks for this, but I am worried
> that any use of unsafe.Pointer might perhaps override these checks
> completely.
>
> This kind of code does not require explicit pinning. As you note,
> passing a pointer to C implicitly pins the memory that the pointer
> points to. Further, the cgo docs say "When passing a pointer to an
> element in an array or slice, the Go memory in question is the entire
> array or the entire backing array of the slice." In other words, yes,
> the slice's entire backing array is implicitly pinned. The cgo tool
> is smart enough to ignore type conversions when deciding what memory
> needs to be pinned.
>
> Ian
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/6d7799a5-4fd3-4025-b76d-d3dcb9d44090n%40googlegroups.com.
[go-nuts] cgo: passing unsafe.Pointer of Slice without explicit pinning?
Hi Gophers,
I'm trying to use cgo for a C library, specifically a function which takes
a void pointer to a buffer, and I'm trying to pass a Go-Slice as this
buffer.
```
func Write(buf []byte) (int, error) {
// var pin rumtime.Pinner
// pin.Pin([0])
// defer pin.Unpin()
// is this necessary? Is it even safe?
...
ret := C.lib_read(unsafe.Pointer([0]), C.int(len(buf))) // function
which reads from this buffer
...
}
```
Does this require explicit pinning of buf? And if so, is the commented part
a valid pinning of a slice? The documentation states that pointers passed
to C functions are implicitly pinned (on the top-level that is), but does
this also apply to slices, i.e. if I pass a (unsafe) pointer to the first
element, is the slice's entire backing array implicitly pinned?
I know that there are some runtime checks for this, but I am worried that
any use of unsafe.Pointer might perhaps override these checks completely.
Thanks a lot.
Julio
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/d3bf9a5d-50dd-4c1c-9f5e-4f7d3f7a3d0cn%40googlegroups.com.
[go-nuts] cgo: Special case in unpinned go pointer checks
Dear golang nuts,
I was trying to gain a better understanding of how cgo works internally,
when I stumbled upon this implementation detail:
https://github.com/golang/go/blob/8db131082d08e497fd8e9383d0ff7715e1bef478/src/runtime/cgocall.go#L628
```
case kindStruct:
st := (*structtype)(unsafe.Pointer(t))
if !indir {
if len(st.Fields) != 1 {
throw("can't happen")
}
cgoCheckArg(st.Fields[0].Typ, p,
st.Fields[0].Typ.Kind_ == 0, top, msg)
return
}
```
This is inside the function where an argument passed to C is checked for
whether it is/contains a pointer to memory containing unpinned go pointers.
In this specific switch-case, the type of the object being checked is a
struct (t.Kind_ is kindStruct). Furthermore,
this is the !indir-branch, which indicates that p is actually the struct
itself and not a pointer to it, i.e. the entire struct is stored within p,
and the struct has only one field. My question is specifically about the
the recursive call
`cgoCheckArg(st.Fields[0].Typ, p, st.Fields[0].Typ.Kind_ ==
0, top, msg)`
I don't understand why it should be possible here for the third ("indir")
argument to be true? What would be a case in which this being set to true
is not contradictory?
When the structs' field's type is not a kindDirectIface, the recursive call
is performed with arguments (st.Fields[0].Typ, p, true, ...). That means,
we are indicating that p is a *pointer to* a value of type
st.Fields[0].Typ. But in the current, enclosing call of the function, we
are in the "!indir"-branch, meaning that p is *exactly* the struct value,
which is exactly the value of its first (and only) field. So we know p is
semantically of type st.Fields[0].Typ, not of type "pointer to
st.Fields[0].Typ". Isn't this a contradiction? How can p be both a pointer
to a st.Fields[0].Typ and be equal to a struct whose only field has type
st.Fields[0].Typ?
The only explanation I can see is that either
st.Fields[0].Typ.Kind_ == 0 is (almost) always false in
practice, and this line catches a very special case, or that there are
cases in which st.Fields[0].Typ contains the value "type Bar" even though
the actual field's value is semantically "pointer to Bar" and not "Bar".
I apologize for the admittedly extremely specific question. I would really
appreciate if someone who is knowledgable about the internals could have a
look at this.
Thanks a lot for your time.
Best Regards,
JC
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/7c63b691-a37d-46ae-8843-3d2177ecd3e1n%40googlegroups.com.
Re: [go-nuts] Is it safe to keep noCopy types in a slice which might copy on re-alloc?
Thanks for your reply, Ian.
I've decided on using a []*Foo now. Just to clarify, however: In general,
copying/assigning from a struct literal as in
a[i] = Foo{}, is that always safe for these sync types? Or generally, is
copying their "untouched" zero-values always safe?
Antonio
On Sunday, August 27, 2023 at 7:05:15 PM UTC+2 Ian Lance Taylor wrote:
> On Sun, Aug 27, 2023 at 7:54 AM Antonio Caceres Cabrera
> wrote:
> >
> > Go vet complains about this minimal example code
> >
> > type Foo struct {
> > val atomic.Uint64
> > }
> >
> > func main() {
> > var foos = make([]Foo, 0)
> > var bar Foo
> > bar.val.Store(5)
> > foos = append(foos, bar) // call of append copies lock value:
> example.com/foo.Foo contains sync/atomic.Uint64 contains
> sync/atomic.noCopy
> >
> > }
> >
> > Because the types of the sync package are not supposed to be copied.
> > This is also true for sync.Map, sync.Mutex, sync.WaitGroup etc.
> >
> > However, if I instead copy in a zero-value Foo and then set that, go vet
> does not complain, even if later appends are done:
> >
> > func main() {
> > var foos = make([]Foo, 0)
> > foos = append(foos, Foo{})
> > foos[0].val.Store(5)
> > foos = append(foos, Foo{})
> > foos = append(foos, Foo{})
> > // append some more
> > // ...
> > fmt.Println(foos)
> > }
> >
> > However, is this supposed to be safe, or is go vet just not catching it?
> > Even if copying a zero-value of such a type is safe, append() might have
> to re-allocate the underlying array, which includes copying the existing
> values, which might already have been used, as foo[0] has been in this
> case. This would violate the noCopy trait of the type. My question is thus:
> >
> > Is it safe to keep nocopy types from the sync and sync.atomic packages
> in a slice, given that they might internally be copied when the slice is
> appended to?
> >
> > Of course, copying these types is not by itself atomic or synchronized
> with other accesses. So let's assume that I, as the programmer, guarantee
> that while an append happens, this slice will only be accessed by the one
> appending goroutine, using other synchronization primitives, such as mutex,
> channels etc. or by only running these appends in the main before any other
> go-routine is started.
>
> Doing an append on a slice of types that contain atomic types is in
> general not safe and go vet is not catching it.
>
> If your program can guarantee that while the append is happening there
> is no concurrent access to the atomic types, then I think that is
> safe. It's not safe in general for all nocopy types, but it's safe
> for types like atomic.Uint64. It doesn't seem like a very clear way
> to write a program though. It may be better to use a slice of
> pointers: in your example, []*Foo. Or precount the size of the slice
> to ensure that append won't copy it.
>
> Ian
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/6ad297ba-19d4-4eaf-a2c2-252ca9db390cn%40googlegroups.com.
[go-nuts] Is it safe to keep noCopy types in a slice which might copy on re-alloc?
Go vet complains about this minimal example code
type Foo struct {
val atomic.Uint64
}
func main() {
var foos = make([]Foo, 0)
var bar Foo
bar.val.Store(5)
foos = append(foos, bar) // call of append copies lock value:
example.com/foo.Foo contains sync/atomic.Uint64 contains sync/atomic.noCopy
}
Because the types of the sync package are not supposed to be copied.
This is also true for sync.Map, sync.Mutex, sync.WaitGroup etc.
However, if I instead copy in a zero-value Foo and then set that, go vet
does not complain, even if later appends are done:
func main() {
var foos = make([]Foo, 0)
foos = append(foos, Foo{})
foos[0].val.Store(5)
foos = append(foos, Foo{})
foos = append(foos, Foo{})
// append some more
// ...
fmt.Println(foos)
}
However, is this supposed to be safe, or is go vet just not catching it?
Even if copying a zero-value of such a type is safe, append() might have to
re-allocate the underlying array, which includes copying the existing
values, which might already have been used, as foo[0] has been in this
case. This would violate the noCopy trait of the type. My question is thus:
Is it safe to keep nocopy types from the sync and sync.atomic packages in a
slice, given that they might internally be copied when the slice is
appended to?
Of course, copying these types is not by itself atomic or synchronized with
other accesses. So let's assume that I, as the programmer, guarantee that
while an append happens, this slice will only be accessed by the one
appending goroutine, using other synchronization primitives, such as mutex,
channels etc. or by only running these appends in the main before any other
go-routine is started.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/31e8d7c2-9198-4ea7-99c9-ed56e13d32c3n%40googlegroups.com.
[go-nuts] Is atomic.CompareAndSwap a read-aquire (even if it returns false)?
Hi Gophers, I have a question about a more subtle part of the language, the memory model regarding atomic operations. The memory model doc on go.dev states: >If the effect of an atomic operation *A* is observed by atomic operation *B*, then *A* is synchronized before *B*.[...] I.e. "observing side-effects" by another atomic operation establishes a synchronizes-before and therefore a happen-before relationship between the write and the observation of that side-effect. My question is basically: Do only atomic *Loads* count as auch an "observation" or does CompareAndSwap also count here? Is there a happens-before between a Store and a later CompareAndSwap (independent of whether the CAS returns false or true?). I asked myself this because the documentation also makes a comparison to C++ and Java: >The preceding definition has the same semantics as C++’s sequentially consistent atomics and Java’s volatile variables. However, there is a small difference between the C++ CAS (compare_exchange) and Go's. In C++, if the swap fails (because the value of the atomic is not the provided expected old value), it will change the old value (which it receives as a reference) to the new value of the atomic. This makes it clear that it counts as a read in any case. In Go (and other languages, e.g. Java), CompareAndSwap does not do this. Thus, one could get the idea that maybe CompareAndSwap doesn't count as a "read" as far as memory model and compiler reorderings and so on are concerned, because it does not read the current value of the atomic on a "program level", returning nothing to the "user", it only reads (compares to) it internally in its inner workings. My question is thus: Does atomic.CompareAndSwap count as a read in this context and is there a happens-before relationship between a previous store and a later compare-and-swap (both if the CAS fails or succeeds?) Thank you all for reading this wall of text. -Antonio -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/633df394-27f1-4416-9602-3c09ade78fban%40googlegroups.com.
