Re: [go-nuts] Re: What are the options to fake SQL database for testing?

[Andrew Werner]

Another choice to throw into the mix is
https://github.com/cockroachdb/copyist. It comes with different trade offs
but if you buy into its framework, it should be much faster than running
the database.

On Thu, Jul 29, 2021 at 3:13 AM Brian Candler  wrote:

> You might also want to look at podman, which runs containers directly as
> processes under your own userid with no docker daemon - and buildah, the
> corresponding tool for creating container images.  I use them quite a lot
> for building and running container images before deploying them to
> kubernetes.
>
> I haven't used the go bindings, but apparently they do exist:
> https://podman.io/blogs/2020/08/10/podman-go-bindings.html
>
> According to that, they are running podman as a service with a REST API,
> which isn't a million miles from a docker daemon - except that the podman
> service runs as your own userid, not as root.
>
> On Thursday, 29 July 2021 at 06:23:05 UTC+1 amits...@gmail.com wrote:
>
>> On Thu, Jul 29, 2021 at 4:39 AM Marcin Romaszewicz 
>> wrote:
>> >
>> > I have this exact testing issue at my company, we have many Go services
>> which use Postgres in production, but are unit tested against SQLite.
>> >
>> > The latest SQLite covers the vast majority of Postgres queries, so most
>> tests simply use an SQLite in-memory DB.
>> >
>> > For the tests which require Postgres- specific functionality, such as
>> partitioned tables, for example, we use
>> https://github.com/testcontainers/testcontainers-go. This is a library
>> which talks to Docker and can create your test prerequisites as docker
>> containers and gives you connection information once they're up. This makes
>> unit tests incredibly slower, but at least functional.
>> >
>> > The danger with mocking too much is that your unit tests end up testing
>> the mocks, and not anything remotely like your runtime environment, so
>> we've chosen to mock as little as possible.
>>
>> Thank you for sharing about testcontainers-go. I have come across
>> testcontainers, it sounds like a viable solution as well. As with a
>> lot of things in software, it seems like we just have to see what
>> works for "us".
>>
>> >
>> > -- Marcin
>> >
>> > On Wed, Jul 28, 2021 at 4:09 AM Henry  wrote:
>> >>
>> >> On Wednesday, July 28, 2021 at 3:05:43 PM UTC+7 amits...@gmail.com
>> wrote:
>> >>>
>> >>> That sounds interesting - is the tool generating or is able to
>> >>> generate SQL for different databases? That must have been a pretty
>> big
>> >>> effort to create such an abstraction.
>> >>>
>> >>
>> >> It produces different SQL for different databases. It supports a
>> limited number of databases. Note that quite a number of Go ORM libraries
>> already support multiple databases. So it is not new. The difference is
>> that other ORM libraries usually provide a general purpose data access
>> library, whereas ours generates more specific codes. Other than that, they
>> serve a similar purpose.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> Groups "golang-nuts" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> an email to golang-nuts...@googlegroups.com.
>> >> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/9c81746a-4fb4-4fb5-8e5f-605169a3f2afn%40googlegroups.com.
>>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "golang-nuts" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to golang-nuts...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29Lv83-4yDijNmukf0Vx%2BoBVZXJPR11bqA_B5CY1mNhOowA%40mail.gmail.com.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/4d6f1b9d-ad44-4fc5-b309-beba2a199382n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BvRuzPvz%2Bu2NCubgnWEiu-D29HVj13tGxUHmEVpaTTdcqyO0Q%40mail.gmail.com.

[go-nuts] unexpected fault address 0xfffffff800000019

[Andrew Werner]

A situation has arisen a few times lately where we've seen segfaults that
have strange looking address values. Another thing that makes the situation
confusing is that the memory referenced by the stack trace should have been
allocated at init time. This seems to only be happening on macOS. The code
in question was built using go1.15.5 on x86_64-apple-darwin20.3.0. We don't
have a reliable repro but it has happened more than once. A user did report
seeing an odd message involving `g->` or so they claim. From my sleuthing,
the only code which could throw that message would be:
https://github.com/golang/go/blob/go1.15.5/src/runtime/proc.go#L703.

Anyway, this all smells of memory corruption to me but I would love
somebody to confirm my feeling that that fault address feels bogus. Is
there any non-corruption reason why one might see this sort of fault?

unexpected fault address 0xfff80019
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1
addr=0xfff80019 pc=0x5658770]

goroutine 2093790 [running]:
runtime.throw(0x865aaf3, 0x5)
/usr/local/opt/go/libexec/src/runtime/panic.go:1116 +0x72
fp=0xc0338bdd18 sp=0xc0338bdce8 pc=0x403c7b2
runtime.sigpanic()
/usr/local/opt/go/libexec/src/runtime/signal_unix.go:749 +0x3e5
fp=0xc0338bdd48 sp=0xc0338bdd18 pc=0x40532e5

Cockroach is tracking this in
https://github.com/cockroachdb/cockroach/issues/62283.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BvRuzNT1-chwnfje0P5as3QGR7ehtuCikR_Pfb6%3D21LqYaHmA%40mail.gmail.com.

Re: [go-nuts] Go PostgreSQL? pg versus pgx

[Andrew Werner]

V4 clearly depends in some form or another on the earlier version. This on
its face also is not necessarily bad. At the end of the day here, we’re
talking about an absolutely tiny amount of data. I hear that you’re new. I
hope I’m successful in setting expectations that even 10s of small
transitive dependencies does not indicate a problem and certainly is not
“bloated beyond repair”. What resources are you concerned with? Was the
compile time slow? Is the number of files upsetting? My best guess is
absolutely not! Code depending on other code which is modularity designed
to fit together is a very good thing. Just trust it for now and raise
concerns in the context of problems backed by data. When you start to have
a bad time for real rather than perceived reasons, come back and let us
know.

-Andrew

On Mon, Feb 22, 2021 at 10:09 AM Paul Förster 
wrote:

> Hi Andrew,
>
> > On 22. Feb, 2021, at 16:02, Andrew Werner  wrote:
> >
> > Hi Paul,
> >
> > What makes you say that that is “heavyweight”? For what it’s worth, many
> of those modules used to just be directories inside the pgx library.
> Splitting them out has been good for the ecosystem. In general, I’d
> encourage you not to worry about a small handful of transitive
> dependencies, especially if they are standard “x” libraries and things from
> the library author as is the case here.
> >
> > Best,
> > Andrew
>
> for example go get saying:
> go: downloading github.com/jackc/pgx v3.6.2+incompatible
>
> and downloading much other stuff makes me think it's heavyweight (at least
> on the modcache). Why is it downloading v3 stuff when I explicitly stated
> v4 in my go get command?
>
> Maybe I'm wrong, maybe I just haven't grasped the module concept fully
> yet, I don't know. That's why I asked. As I said, I'm new to Go and just
> making my first steps beyond simple things.
>
> Cheers,
> Paul

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BvRuzPZ%2BFU5%2BgVnYECq4TezbY3zOOT2NioiT067COPkVT1AYw%40mail.gmail.com.

Re: [go-nuts] Go PostgreSQL? pg versus pgx

[Andrew Werner]

Hi Paul,

What makes you say that that is “heavyweight”? For what it’s worth, many of
those modules used to just be directories inside the pgx library. Splitting
them out has been good for the ecosystem. In general, I’d encourage you not
to worry about a small handful of transitive dependencies, especially if
they are standard “x” libraries and things from the library author as is
the case here.

Best,
Andrew

On Mon, Feb 22, 2021 at 9:56 AM Paul Förster 
wrote:

> Hi Matthew,
>
> > On 22. Feb, 2021, at 15:28, Matthew Holiday 
> wrote:
> >
> > From the "status" section at the bottom of the README for pq:
> >
> > "This package is effectively in maintenance mode and is not actively
> developed. Small patches and features are only rarely reviewed and merged.
> We recommend using pgx which is actively maintained."
>
> in this case, can I reduce the heavyweight somehow? Why would I need all
> this stuff? I only want to be able to connect and do a few selects, no DML,
> no DDL, just a few select statements. Also, I'm a fan of small footprints.
> :-)
>
> I'm new to Go, so please bear with me if my question seems somewhat
> strange to you.
>
> Cheers,
> Paul
>
>
> $ go get github.com/jackc/pgx/v4
> go: downloading github.com/jackc/pgx/v4 v4.10.1
> go: downloading github.com/jackc/pgx v3.6.2+incompatible
> go: downloading github.com/jackc/pgio v1.0.0
> go: downloading github.com/jackc/pgconn v1.8.0
> go: downloading github.com/jackc/pgproto3 v1.1.0
> go: downloading github.com/jackc/pgtype v1.6.2
> go: downloading golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
> go: downloading github.com/jackc/chunkreader v1.0.0
> go: downloading github.com/jackc/pgpassfile v1.0.0
> go: downloading github.com/jackc/pgservicefile
> v0.0.0-20200714003250-2b9c44734f2b
> go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
> go: downloading github.com/jackc/pgproto3/v2 v2.0.6
> go: downloading golang.org/x/text v0.3.3
> go: downloading github.com/jackc/chunkreader/v2 v2.0.1
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/B0F53786-F171-42DE-9413-8B54B99C2732%40gmail.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BvRuzOgRRzdvEVV9DiPa-0cA9bjAtH89e2EGEh1wjd1%2B2omPA%40mail.gmail.com.

Re: [go-nuts] Block-based data structures and the Type Parameters - Draft Design

[Andrew Werner]

Hey Roger,

Thanks for taking the time to engage with me on this. Really appreciate it!


On Monday, June 22, 2020 at 6:01:46 PM UTC-4, rog wrote:
>
>
>
> On Mon, 22 Jun 2020 at 22:49, Andrew Werner  > wrote:
>
>>
>>
>> On Monday, June 22, 2020 at 5:42:12 PM UTC-4, rog wrote:
>>>
>>>
>>> Thanks for pointing this out. I'm cool with this approach. I'll update 
>>>>> my library to utilize it (and consider also adopting the list.List, 
>>>>> though 
>>>>> I do like my freedom to pool list nodes).
>>>>>
>>>>
>>> Personally, I'd start by re-using list.List, and only use pools if there 
>>> really is a significant performance benefit to be gained there.
>>> If you just need zero-valued elements, I could imagine that sync.Pool 
>>> might end up slower than just using the GC.
>>>
>>
>> Fair. 
>>
>>
>>> Hopefully with aggressive inlining of the specialized type the compiler 
>>>>> will be able to make the optimizations that it would be able to make if 
>>>>> the 
>>>>> slicing were performed directly on an array.
>>>>>
>>>>
>>> What optimisations are you thinking of here? Are you concerned that 
>>> slice operations are too slow?
>>>
>>
>>
>> * Bound check elimination
>>
>
> I'm not sure how it could do that, given that it's inevitable that an 
> index is stored in a field with no range limits on it (i guess if you used 
> a 256-element array and a uint8 index, it could, but then it wouldn't work 
> for other array sizes).
>
Fair point. I imagine that the fact that it's an array doesn't really 
change much w.r.t. when a bounds check needs to occur, it just might change 
the shape of such a check. It does seem like in my ring buffer example use 
of unsigned integers in some places can eliminate some bounds checking 
given I'm going to mod the whole thing by `len(buf)`

I do recognize that the above case has nothing to do with the slice/array 
discussion.



type ringBuf(type T, *ArrayT Slicer(T)) struct {
head, len uint32
Array
}

func (rb *ringBuf(T, ArrayT)) at(idx int32) *T {
buf := rb.Slice()
// no bounds check needed.
return [int(rb.head+idx) % len(buf)]
}





> * Avoiding an indirection through the pointer in the slice header would be 
>> nice
>>
>
> Isn't it going to have to indirect through a pointer regardless of whether 
> it's a slice or an array? The array isn't stored inside the Dequeue struct 
> itself.
>

In your example where you put the slice into the dequeue itself, yes, sure. 
I used the Dequeue as a simplified example where the motivating case is 
really a B-Tree. In the B-tree we'd instead be operating at a node basis 
where the node does actually contain the array. In that case we either 
realize a slice on the stack or realize a slice in the node to refer to 
itself. In either case, there will be an indirection through the slice 
header. I'll buy the argument that it should be cheap and hit cache lines.
 

>
> * Compile-time evaluation of len and cap
>>
>
> Isn't that just a one-word load of memory that's very likely in the same 
> cache line as the array-pointer member?
> I can't see that this is likely to make a significant performance 
> difference, but I'd love to see some numbers.
>
Fair enough. Anything we do here is going to be better than a world today. 
I'm just trying to push on the idea that it will be a bummer if after go 
generics land, people in the performance critical space still find 
themselves using code-generating generics solutions like 
https://github.com/mmatczuk/go_generics to specialize data structures. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9171440e-98a8-445d-b374-2e31e89b0abeo%40googlegroups.com.

Re: [go-nuts] Block-based data structures and the Type Parameters - Draft Design

[Andrew Werner]

On Monday, June 22, 2020 at 5:42:12 PM UTC-4, rog wrote:
>
>
> Thanks for pointing this out. I'm cool with this approach. I'll update my 
>>> library to utilize it (and consider also adopting the list.List, though I 
>>> do like my freedom to pool list nodes).
>>>
>>
> Personally, I'd start by re-using list.List, and only use pools if there 
> really is a significant performance benefit to be gained there.
> If you just need zero-valued elements, I could imagine that sync.Pool 
> might end up slower than just using the GC.
>

Fair. 


> Hopefully with aggressive inlining of the specialized type the compiler 
>>> will be able to make the optimizations that it would be able to make if the 
>>> slicing were performed directly on an array.
>>>
>>
> What optimisations are you thinking of here? Are you concerned that slice 
> operations are too slow?
>


* Bound check elimination
* Avoiding an indirection through the pointer in the slice header would be 
nice
* Compile-time evaluation of len and cap

IIUC all of these would happen if we were operating on a raw array type 
rather than a slice. 


>   cheers,
> rog.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6a6bd9d6-deec-42f2-a9d8-7a1413221f4do%40googlegroups.com.

Re: [go-nuts] Block-based data structures and the Type Parameters - Draft Design

[Andrew Werner]

On Monday, June 22, 2020 at 10:17:01 AM UTC-4, rog wrote:
>
>
>
> On Mon, 22 Jun 2020 at 14:59, Andrew Werner  > wrote:
>
>>
>>
>> On Mon, Jun 22, 2020 at 9:45 AM roger peppe > > wrote:
>>
>>>
>>>
>>> On Mon, 22 Jun 2020 at 14:26, Andrew Werner >> > wrote:
>>>
>>>> Hey Rog,
>>>>
>>>> I think I sent you a private reply earlier. My bad. I now see what 
>>>> you're proposing in the first proposal think it makes a lot of sense.
>>>>
>>>> The thing that I think I had missed was this piece of magic:
>>>> // New returns a new Dequeue instance using B as the block storage 
>>>> backing.
>>>> func New(type T interface{}, *B Block(T))() *Dequeue(T) {
>>>>
>>>> One concern I have here is that the `Slice` method is on the array 
>>>> itself so it's possible that invoking it will copy the entire array to the 
>>>> stack. Does that concern you?
>>>>
>>>
>>> It won't copy the array because it's a pointer method (that's what the 
>>> "*B" means).
>>> In fact it *must not* copy the array because then it couldn't return a 
>>> reference to it.
>>>
>>
>> Oh! It’s saying that *B implements the constraint, nifty. Is this idea in 
>> the current proposal?
>>
>
> Yes. See 
> https://go.googlesource.com/proposal/+/refs/heads/master/design/go2draft-type-parameters.md#pointer-methods
>  
>
>> I agree that it’ll do the trick in the same way as your second proposal 
>> and my proposal and is even a bit cleaner. That being said, it still seems 
>> worse than having a mechanism to indicate that a type really is an array 
>> and can be syntactically used as such, no? 
>>
>
> I'm not sure. ISTM that there is a reasonable workaround and that the 
> alternative might significantly complicate the generics proposal. The 
> authors aren't trying to address every possible generics use case, 
> especially when it's possible to work around the problem, and I like that 
> approach :)
>

Cool, I'm on board with this view. I'll adopt this approach and see how it 
feels. Hopefully with aggressive inlining the trip through the compiler 
will be able to optimize the access as though the array were being indexes 
directly.
 

>
> I suppose if the Slice method gets reasonably inclined the compiler could 
>> do good things. 
>>
>> I think I’m in favor of this proposal in addition to my proposal that 
>> allows a type to be declared as an array and to thus be used syntactically 
>> as an array. 
>>
>>
>>> The second proposal feels almost identical to what I proposed. Can you 
>>>> help me understand how it differs? 
>>>>
>>>
>>> It is indeed a similar approach. The main difference (beyond the 
>>> somewhat simpler dequeue implementation itself) is that my New method 
>>> returns a concrete type, not an interface. This means that it's easier to 
>>> jump to the definition of its method implementations, potentially more 
>>> performant because it can use direct rather than indirect calls (*), and 
>>> arguably more idiomatic Go. It also means it's possible to add more methods 
>>> without breaking backward compatibility, which is a significant benefit in 
>>> my view.
>>>
>>>   cheers,
>>> rog.
>>>
>>> (*) you'll still get *some* indirect calls, amortised over the buffer 
>>> size.
>>>
>>
>> Ack, sure, thanks for clarifying. No quibbles from me on this front. 
>>
>> FWIW the Alloc* methods came from some experience with these data 
>> structures where I want to decode a stream of structs off the network or 
>> something so I want pointers to zero values to then pass to Decode or 
>> Unmarshal. I’ll admit it feels weird without the context.  
>>
>
> Note that with the concrete type return, it would be possible to add 
> Alloc* methods at a later stage without breaking backward compatibility.
>
> As noted in a comment in my code, I think that it would be nice to add 
> similar methods to the list package too - they wouldn't have been useful 
> before with the interface{} value approach, but definitely are now (for 
> example you could have an in-element mutex).
>
>   cheers,
> rog.
>
>   
>
>>  
>>>
>>>>
>>>> On Monday, June 22, 2020 at 7:58:43 AM UTC-4, rog wrote:
>>>>>
>>>>> Thanks for the interesting use case, Andrew!
>>>>>
>>>

Re: [go-nuts] Block-based data structures and the Type Parameters - Draft Design

[Andrew Werner]

On Mon, Jun 22, 2020 at 9:45 AM roger peppe  wrote:

>
>
> On Mon, 22 Jun 2020 at 14:26, Andrew Werner  wrote:
>
>> Hey Rog,
>>
>> I think I sent you a private reply earlier. My bad. I now see what you're
>> proposing in the first proposal think it makes a lot of sense.
>>
>> The thing that I think I had missed was this piece of magic:
>> // New returns a new Dequeue instance using B as the block storage
>> backing.
>> func New(type T interface{}, *B Block(T))() *Dequeue(T) {
>>
>> One concern I have here is that the `Slice` method is on the array itself
>> so it's possible that invoking it will copy the entire array to the stack.
>> Does that concern you?
>>
>
> It won't copy the array because it's a pointer method (that's what the
> "*B" means).
> In fact it *must not* copy the array because then it couldn't return a
> reference to it.
>

Oh! It’s saying that *B implements the constraint, nifty. Is this idea in
the current proposal? I agree that it’ll do the trick in the same way as
your second proposal and my proposal and is even a bit cleaner. That being
said, it still seems worse than having a mechanism to indicate that a type
really is an array and can be syntactically used as such, no?

I suppose if the Slice method gets reasonably inclined the compiler could
do good things.

I think I’m in favor of this proposal in addition to my proposal that
allows a type to be declared as an array and to thus be used syntactically
as an array.


> The second proposal feels almost identical to what I proposed. Can you
>> help me understand how it differs?
>>
>
> It is indeed a similar approach. The main difference (beyond the somewhat
> simpler dequeue implementation itself) is that my New method returns a
> concrete type, not an interface. This means that it's easier to jump to the
> definition of its method implementations, potentially more performant
> because it can use direct rather than indirect calls (*), and arguably more
> idiomatic Go. It also means it's possible to add more methods without
> breaking backward compatibility, which is a significant benefit in my view.
>
>   cheers,
> rog.
>
> (*) you'll still get *some* indirect calls, amortised over the buffer
> size.
>

Ack, sure, thanks for clarifying. No quibbles from me on this front.

FWIW the Alloc* methods came from some experience with these data
structures where I want to decode a stream of structs off the network or
something so I want pointers to zero values to then pass to Decode or
Unmarshal. I’ll admit it feels weird without the context.


>
>>
>> On Monday, June 22, 2020 at 7:58:43 AM UTC-4, rog wrote:
>>>
>>> Thanks for the interesting use case, Andrew!
>>>
>>> I've experimented with a slightly different approach:
>>>
>>> https://go2goplay.golang.org/p/AkqzbWmpj6t
>>>
>>> It expects the caller to implement a method on the array to get a
>>> reference to the
>>> underlying storage, but that's fairly trivial to implement.
>>>
>>> I've built it on top of the "list" package from the stdlib (available in
>>> the dev.go2go branch under src/cmd/go2go/testdata/go2path).
>>>
>>> I think it has the potential to be a bit more performant as the
>>> type-hiding abstraction (Block in my example) is inside the static type, so
>>> most calls can be direct rather than indirect.
>>>
>>> It's *almost* possible to make this implementation use a slice-based
>>> implementation of Block too (thus allowing dynamically specified block
>>> sizes at the cost of an extra indirection through the linked list node).
>>> That's not possible because the pointer method is invoked on the zero-value
>>> of the type, so without using a global variable, there's no way for the
>>> method to find out the required block size. A way to work around that is to
>>> use another level of indirection instead of using pointer methods directly:
>>>
>>> https://go2goplay.golang.org/p/nMreEMpqXd1
>>>
>>> This makes me wonder whether the pointer methods restriction in the
>>> generics draft proposal might not be unnecessary and perhaps not
>>> necessarily a good idea. ISTM that the only reason for restricting methods
>>> to pointer-only is so that they can be called on zero values, and that
>>> necessarily restricts the available parameterisation to static-only (or
>>> using global variables). I suspect that might turn out to be an
>>> anti-pattern. Instead, using function or interface value that operates on
>>> the pointer is strictly more general, I think, and

Re: [go-nuts] Block-based data structures and the Type Parameters - Draft Design

[Andrew Werner]

Hey Rog,

I think I sent you a private reply earlier. My bad. I now see what you're 
proposing in the first proposal think it makes a lot of sense.

The thing that I think I had missed was this piece of magic:
// New returns a new Dequeue instance using B as the block storage backing.
func New(type T interface{}, *B Block(T))() *Dequeue(T) {

One concern I have here is that the `Slice` method is on the array itself 
so it's possible that invoking it will copy the entire array to the stack. 
Does that concern you?

The second proposal feels almost identical to what I proposed. Can you help 
me understand how it differs? 

On Monday, June 22, 2020 at 7:58:43 AM UTC-4, rog wrote:
>
> Thanks for the interesting use case, Andrew!
>
> I've experimented with a slightly different approach:
>
> https://go2goplay.golang.org/p/AkqzbWmpj6t
>
> It expects the caller to implement a method on the array to get a 
> reference to the
> underlying storage, but that's fairly trivial to implement.
>
> I've built it on top of the "list" package from the stdlib (available in 
> the dev.go2go branch under src/cmd/go2go/testdata/go2path).
>
> I think it has the potential to be a bit more performant as the 
> type-hiding abstraction (Block in my example) is inside the static type, so 
> most calls can be direct rather than indirect.
>
> It's *almost* possible to make this implementation use a slice-based 
> implementation of Block too (thus allowing dynamically specified block 
> sizes at the cost of an extra indirection through the linked list node). 
> That's not possible because the pointer method is invoked on the zero-value 
> of the type, so without using a global variable, there's no way for the 
> method to find out the required block size. A way to work around that is to 
> use another level of indirection instead of using pointer methods directly:
>
> https://go2goplay.golang.org/p/nMreEMpqXd1
>
> This makes me wonder whether the pointer methods restriction in the 
> generics draft proposal might not be unnecessary and perhaps not 
> necessarily a good idea. ISTM that the only reason for restricting methods 
> to pointer-only is so that they can be called on zero values, and that 
> necessarily restricts the available parameterisation to static-only (or 
> using global variables). I suspect that might turn out to be an 
> anti-pattern. Instead, using function or interface value that operates on 
> the pointer is strictly more general, I think, and doesn't suffer from that 
> problem.
>
>   cheers,
> rog.
>
>
> On Sun, 21 Jun 2020 at 17:43, Andrew Werner  > wrote:
>
>> Thanks for the reply! If I read it correctly, it is already possible to 
>> specify a type constraint as an Array of a given size and so the first part 
>> of the "What I might have expected" actually does work. I noticed that it 
>> works in that it works as a type constraint but I found that types 
>> constrained that way do not support indexing or slicing. Perhaps that's 
>> just a limitation of the prototype and not the proposal.
>>
>> The other thing that I think I'm proposing (thanks for the correction) is 
>> a way to represent all arrays of a given type:
>>
>> type Array(type T) interface {
>> type [...]T
>> }
>>
>> Thanks for dealing with my very long-winded way of saying that. 
>>
>> On Sunday, June 21, 2020 at 12:35:53 PM UTC-4, David Finkel wrote:
>>>
>>>
>>>
>>> On Sat, Jun 20, 2020 at 1:22 AM Andrew Werner  wrote:
>>>
>>>> [The body of this email is a duplication of the README in 
>>>> https://github.com/ajwerner/go2dequeue/ 
>>>> which also contains the sample implementation]
>>>>
>>>> Exercise building a dequeue with the go2 Type Parameter Draft 
>>>>
>>>> This project is an exploration with the go2go / Type Parameters - 
>>>> Design Draft 
>>>> <https://go.googlesource.com/proposal/+/refs/heads/master/design/go2draft-type-parameters.md>
>>>>  
>>>> with an eye towards block-based data structures which promote access 
>>>> locality.
>>>>
>>>> Such data structures traditionally utilize blocks of objects laid out 
>>>> contiguously in memory. In go, today, one generally specializes such data 
>>>> structures which are performance critical. This is especially important in 
>>>> the case of reducing allocations which occur on critical paths. Using a 
>>>> sync.Pool can help with allocations in some cases but can be clunky to use 
>>>> and, well, do create more pointers for GC to worry about and don't have 
>

[go-nuts] Re: [generics] Zero value

[Andrew Werner]

In order to get a zero value, does the following work today? Is it 
acceptable?

func GenericFunc(type T)() T {
return func() (t T) { return t }()
}

This could also just be written like below but above I was trying to 
demonstrate how to make a zero value within another function.

func ZeroValue(type T)() (t T) {
return t
}



On Monday, June 22, 2020 at 8:09:11 AM UTC-4, Max wrote:
>
> Types are not first-class in Go, thus T.someMethod() is somewhat an 
> unusual syntax.
>
> The proposal  to use 'T{}' to 
> mean zero value of T breaks Go compatibility promise:
> if T is a map type or a slice type, 'T{}' already has a meaning:
> create a non-nil, zero-elements map (or slice).
> Instead the zero value of such types is nil.
>
> Using '_' could be an elegant solution in places where type inference can 
> deduce the correct type.
>
> Two more ideas that I did not see yet are:
> 1. use 'T()' to mean zero value of T - currently does not compile, thus Go 
> compatibility promise is preserved.
> 2. define a new compiler builtin 'zero(T)'
>
> On Friday, June 19, 2020 at 1:59:59 PM UTC+2, Ivan Ivanyuk wrote:
>>
>> What about something like int.CreateZeroValue(), or T.CreateZeroValue() 
>> and CreateZeroValueInterface?
>>
>> On Thursday, June 18, 2020 at 7:34:55 PM UTC+3, bruno...@gmail.com wrote:
>>>
>>> First, congratulations on the working compiler! Very cool to try out 
>>> generics in Go.
>>>
>>> I was experimenting  with 
>>> some simple generic functions and felt the need for the zero value of a 
>>> generic type. I've read the discussion anticipated in your proposal 
>>> ,
>>>  
>>> so I'd like to use this topic just to provide my sentiment, and perhaps 
>>> gather it from others.
>>>
>>> In my opinion, the most Go-like proposal 
>>>  is using '_' to signify the 
>>> zero value in RHS. We are already used to having '_' in LHS to accept 
>>> whatever type, and although the meaning is slightly changed (from 
>>> 'whatever' to 'zero'), it's conceptually close.
>>>
>>> I'm opposed to a change that uses '{}' or 'T{}' to mean zero type, as in 
>>> this proposal , because in 
>>> my mind curly braces are used for composite types. Imagining that 'T' is a 
>>> placeholder, It's weird replacing 'T{}' for 'int{}', for example. I'm super 
>>> in favor of having a leaner syntax for composite types, though .
>>>
>>> Whatever is decided, please don't punt on this issue and recommend 
>>> '*new(T)'. That's really ugly and feels wrong to allocate and immediately 
>>> dereference the value just to circumvent a syntax change. Even if the 
>>> compiler is smart enough not to allocate anything, that's what's written in 
>>> the source code.
>>>
>>> Thanks.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b84a3ece-b44e-46b3-832c-485b4cf1427co%40googlegroups.com.

Re: [go-nuts] Block-based data structures and the Type Parameters - Draft Design

[Andrew Werner]

Thanks for the reply! If I read it correctly, it is already possible to 
specify a type constraint as an Array of a given size and so the first part 
of the "What I might have expected" actually does work. I noticed that it 
works in that it works as a type constraint but I found that types 
constrained that way do not support indexing or slicing. Perhaps that's 
just a limitation of the prototype and not the proposal.

The other thing that I think I'm proposing (thanks for the correction) is a 
way to represent all arrays of a given type:

type Array(type T) interface {
type [...]T
}

Thanks for dealing with my very long-winded way of saying that. 

On Sunday, June 21, 2020 at 12:35:53 PM UTC-4, David Finkel wrote:
>
>
>
> On Sat, Jun 20, 2020 at 1:22 AM Andrew Werner  > wrote:
>
>> [The body of this email is a duplication of the README in 
>> https://github.com/ajwerner/go2dequeue/ 
>> which also contains the sample implementation]
>>
>> Exercise building a dequeue with the go2 Type Parameter Draft 
>>
>> This project is an exploration with the go2go / Type Parameters - Design 
>> Draft 
>> <https://go.googlesource.com/proposal/+/refs/heads/master/design/go2draft-type-parameters.md>
>>  
>> with an eye towards block-based data structures which promote access 
>> locality.
>>
>> Such data structures traditionally utilize blocks of objects laid out 
>> contiguously in memory. In go, today, one generally specializes such data 
>> structures which are performance critical. This is especially important in 
>> the case of reducing allocations which occur on critical paths. Using a 
>> sync.Pool can help with allocations in some cases but can be clunky to use 
>> and, well, do create more pointers for GC to worry about and don't have the 
>> access locality.
>>
>> Whether it's really important to have data structures which include 
>> arrays of other data structures laid out contiguously in RAM is really 
>> important is sort of orthogonal. Let's just assume that it is for now and 
>> look at how we'd do it. We only need to consider rejecting the importance 
>> of this case if we can't find a good solution or idiom to support it. The 
>> google/btree <https://github.com/google/btree> library README links this 
>> Google 
>> Open Source Blog Post on block-based container 
>> <https://opensource.googleblog.com/2013/01/c-containers-that-save-memory-and-time.html>
>>  
>> performance indicating it does matter. The interesting thing about that 
>> library is it hardly gets the access locality of the C++ library in the 
>> blog post it references.
>>
>> The challenge this library explores is to layout blocks in structures 
>> contiguously in RAM while allowing the user to have some control over that 
>> block size.
>> This example 
>>
>> The approach this experiment takes is to allow the users to specify the 
>> block size by providing a function to map an array type to a slice. The 
>> allows the blocks to contain a slice which references the array. The 
>> overhead to maintain the slice header is 3 words but the cost is probably 
>> less in memory and more in the optimizations the compiler will be less able 
>> to perform. In particular, it probably will need to do bounds checking on 
>> that slice and there are probably some opportunities to avoid the pointer 
>> interactions. The interface ends up being pretty reasonable though a little 
>> bit ... opaque?
>>
>> What this looked like in this demo is*
>>
>> // Say we want to create a dequeue of things.type Thing struct { Foo int64, 
>> Bar int64, ID [16]byte }// The below incantation will create one with blocks 
>> of 37 elements.d := NewWithArray(func(a *[37]Thing) []Thing { return (*a)[:] 
>> })// The nodes in the linked list of this structure are 1240 bytes, not 
>> that// that's a great number or anything like that, just saying it's pretty 
>> big.// Maybe there'd be something good to get out of aligning things are 
>> cache line// sizes. See TestNodeSize.
>>
>> Other things you could do Use a slice for the buffer 
>>
>> A different approach would be to just have the array in the node, take a 
>> buffer size and then keep a sync.Pool or freelist of slices. Today's 
>> github.com/google/btree does this but way worse in that it just keeps a 
>> slice of btree.Item. So fine, it's be two objects, the nodes and their item 
>> buffers. Maybe that is the answer but it's pretty lame. My feeling is once 
>> somebody is optimizing a data structure for access locality they are trying 
>> to optimize as far as they 

[go-nuts] Block-based data structures and the Type Parameters - Draft Design

[Andrew Werner]

[The body of this email is a duplication of the README in
https://github.com/ajwerner/go2dequeue/
which also contains the sample implementation]

Exercise building a dequeue with the go2 Type Parameter Draft

This project is an exploration with the go2go / Type Parameters - Design
Draft

with an eye towards block-based data structures which promote access
locality.

Such data structures traditionally utilize blocks of objects laid out
contiguously in memory. In go, today, one generally specializes such data
structures which are performance critical. This is especially important in
the case of reducing allocations which occur on critical paths. Using a
sync.Pool can help with allocations in some cases but can be clunky to use
and, well, do create more pointers for GC to worry about and don't have the
access locality.

Whether it's really important to have data structures which include arrays
of other data structures laid out contiguously in RAM is really important
is sort of orthogonal. Let's just assume that it is for now and look at how
we'd do it. We only need to consider rejecting the importance of this case
if we can't find a good solution or idiom to support it. The google/btree
 library README links this Google Open
Source Blog Post on block-based container

performance indicating it does matter. The interesting thing about that
library is it hardly gets the access locality of the C++ library in the
blog post it references.

The challenge this library explores is to layout blocks in structures
contiguously in RAM while allowing the user to have some control over that
block size.
This example

The approach this experiment takes is to allow the users to specify the
block size by providing a function to map an array type to a slice. The
allows the blocks to contain a slice which references the array. The
overhead to maintain the slice header is 3 words but the cost is probably
less in memory and more in the optimizations the compiler will be less able
to perform. In particular, it probably will need to do bounds checking on
that slice and there are probably some opportunities to avoid the pointer
interactions. The interface ends up being pretty reasonable though a little
bit ... opaque?

What this looked like in this demo is*

// Say we want to create a dequeue of things.type Thing struct { Foo
int64, Bar int64, ID [16]byte }// The below incantation will create
one with blocks of 37 elements.d := NewWithArray(func(a *[37]Thing)
[]Thing { return (*a)[:] })// The nodes in the linked list of this
structure are 1240 bytes, not that// that's a great number or anything
like that, just saying it's pretty big.// Maybe there'd be something
good to get out of aligning things are cache line// sizes. See
TestNodeSize.

Other things you could do Use a slice for the buffer

A different approach would be to just have the array in the node, take a
buffer size and then keep a sync.Pool or freelist of slices. Today's
github.com/google/btree does this but way worse in that it just keeps a
slice of btree.Item. So fine, it's be two objects, the nodes and their item
buffers. Maybe that is the answer but it's pretty lame. My feeling is once
somebody is optimizing a data structure for access locality they are trying
to optimize as far as they can go.
Ignore or augment the language generics with manual or automatic
specialization

Today, in performance critical cases, people use a variety of automatic or
manual specialization. There are tools like
https://github.com/cheekybits/genny or
https://github.com/mmatczuk/go_generics. Note that the latter has been used
to good effect in CockroachDB for building specialized copy-on-write,
interval B-trees.

One answer to this problem might just be that go's generics solution
doesn't solve for the problem of specializing block-based data structures.
That's not a particularly satisfying answer.
What I might have expected / guess I'm proposing

Type lists are a thing for utilizing language features which only apply to
certain kinds of types. There should be a way to create a type list for
arrays of types.

As a first pass I would have anticipated that there would be a way to
inform this package of a buffer size maybe drawn from a fixed number of
sizes. For example:

type BlockSize int
const (
_ BlockSize = 1 << (4*iota)
Block16
Block256
)
// Array here allowstype Array(type T) interface {
type [Block16]T, [Block256]T
}
func New(type T)(blockSize BlockSize) Dequeue(T) {
switch blockSize {
case Block16:
return &(dequeue(T, [Block16]T){ ... })
case Block256:
return &(dequeue(T, [Block16]T){})
}
}
type dequeue(type T interface{}, ArrayT Array(T)) struct {
len
}
type node(type T interface{}, ArrayT Array(T)) struct {
left, right