Re: [go-nuts] Re: slog's use of runtime.Callers() with a skip

['Shaun Crampton' via golang-nuts]

>
> Do you have any evidence to the contrary?


Only that when Go 1.12 dropped, our similar function stopped working and
that reducing the skip seemed to do the trick.

The symptom was that our function would see an assembly file as the caller,
which I interpreted to mean that we'd skipped too far.  It only happened in
goroutines with short stacks and I put it down to inlining.

Quite possible that the "fix" I made was mistaken, or that runtime.Callers
has been updated/fixed since then.

In case you want to stare at my fix, here is the diff:
https://github.com/projectcalico/libcalico-go/commit/1f1ababe294be198148c4232ef6c0344898d3b31#diff-7ab3329a79d456fe0d1747bba7344ac61e839f9097ae65d09b36f3c11ea13fbdL153


Looking back, I see three changes in that "fix":

   - Change to skip from 6 to 1 (and increase pcs buffer size accordingly).
   - Reslice pcs to the valid portion, looks like we missed that before;
   possible this was the "real" fix?
   - Change the list of file names that we skip.

We were already using CallersFrames before the fix.

The Go runtime does the right thing.
>

It does seem to in my local tests with up-to-date Go; i tried some toy
examples and checked that functions really were being inlined.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMhR0U1DtH_HLPFDO%2BRPT1xU0SxynoP_63uNE%2BzJ%2BES9wX%3D2Xg%40mail.gmail.com.

Re: [go-nuts] Syscalls that take structs with embedded pointers

[Shaun Crampton]

> You can use a *byte for the buffer. For instance, unix.Iovec does this

Are you sure that's not a bug?  What's to stop the *Byte from being
moved by the GC?

On Fri, May 14, 2021 at 2:27 PM Michael Pratt  wrote:
>
> You can use a *byte for the buffer. For instance, unix.Iovec does this: 
> https://pkg.go.dev/golang.org/x/sys/unix#Iovec
>
> Users can cast a *unix.Iovec directly to unsafe.Pointer for Syscall without 
> any special handling of the *byte field.
>
> On Fri, May 14, 2021, 09:01 sh...@tigera.io  wrote:
>>
>> Now, is it technically legal to convert a uintptr to some location that was 
>> manually allocated and then cast it to an unsafe.Pointer?  When I look at 
>> the docs for unsafe.Pointer, I can't match that to any of the cases but it 
>> seems very likely to be safe by analogy with the GCO rules for handling "C" 
>> pointers and "Go" pointers.
>>
>> On Friday, May 14, 2021 at 1:24:28 PM UTC+1 sh...@tigera.io wrote:
>>>
>>> Thanks for the pointer to that package; looking at our go.mod, looks like 
>>> we've already got it as a transitive dependency so it looks ideal for my 
>>> use case.
>>>
>>> On Friday, May 14, 2021 at 12:10:45 PM UTC+1 Jan Mercl wrote:

 On Fri, May 14, 2021 at 12:36 PM sh...@tigera.io  wrote:

 > One way I could see to solve this would be to do some manual memory 
 > management with MMap or BRK to allocate the buffer but I wondered if 
 > there was an easier way?

 IMO using manual memory management _is_ the easy way in this case. I'm
 using https://pkg.go.dev/modernc.org/memory for this.
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/317d44cf-8f2b-42d6-ab10-b42da7280616n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAMhR0U2_QWLimoS0GJtwHCjFyJt-0qct3n-N%2BP98n-wdjeZPYw%40mail.gmail.com.

Re: [go-nuts] Re: Struggling with go mod, micro-repos and private forks

[Shaun Crampton]

We have https://golang.org/issue/26232 open for 2FA workflows in general.
>
> In the meantime, you may need to configure a Personal Access Token 
> 
>  to 
> get HTTPS to work.
>


We use 2FA for accessing the github UI but we tend to use SSH for push and 
pull.  Would be easier if it tried ssh rather than bailing out.

 

>
>>- Another team member said they tried using 
>>v0.0.0-0001010100- as version but it got replaced.
>>
>> That probably means that you have a higher version requirement from some 
> other dependency. (The `go` command automatically updates the `go.mod` file 
> to maintain consistency.)
>
>>
>>
But then we go round in circles, right?  The start of this discussion was 
that I wanted a "nil" version to put in my require statement so that I 
could replace it with exactly the commit that I need using a "replace".  
Now, if the tool helpfully updates the require then that defeats the point 
of having a "nil" version in the first place.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7360a2f8-7787-4392-8077-7c4790c55d3e%40googlegroups.com.

[go-nuts] Re: Struggling with go mod, micro-repos and private forks

[Shaun Crampton]

I circed some of the suggestions round my team.  Sounds like others had 
already tried some of the suggestions with mixed results:


   - Go v1.13 still has trouble authenticating to github without an 
   "insteadof" in the config.  We use 2FA on github, which seems to make HTTPS 
   fail in a way that doesn't fall back to git+ssh mode?
   - Another team member said they tried using 
   v0.0.0-0001010100- as version but it got replaced.



On Thursday, October 24, 2019 at 10:35:32 AM UTC+1, Shaun Crampton wrote:
>
>
>> The good news is that we're aware of (and planning to address) most of 
>> these pain points; the bad news is that we haven't been able to get to most 
>> of them yet.
>>
>>
> Great to see, thanks!
>
>  
>
>>
>>> I think the biggest problem we have is when working with private repos.  
>>>  What I want to express to the tool is 
>>>
>>> My module requires commit abcd1234 (or version v1.2.3) of dependency 
>>> x/y/z
>>>
>>> Look for any instances of dependency x/y/z in git repo 
>>> g...@github.com:ourfork instead of the default.
>>>
>>>  
>> I believe this is https://golang.org/issue/28176.
>>
>> It's an intriguing idea, but problematic if the dependencies are 
>> specified as pseudo-versions, because the commit hashes in a fork in 
>> general will not match the hashes in the upstream module. We're still 
>> exploring alternatives for this use-case.
>>
>
>
> So, the problem that pseudo-versions are trying to solve is ordering in 
> the face of unordered commit hashes?  For my use case, the way I want this 
> to work is that, once a dependency is pinned to my private fork, it's as if 
> the public repo didn't exist.  In 99% of cases, all the pins that I'm 
> working with are my organisation's repos anyway so once we pin to a private 
> fork that's what we want to use everywhere, transitively, even in 
> downstream repos.  In extreme cases, even the version numbering scheme in 
> the private fork may be different so there's not necessarily any relation 
> between the open source v2.3.0 and the private v2.3.0 tags.
>
> Our minority use case is forking some public repo to make a fix.  In that 
> case, a perfect tool would have a way to say "we need a version >= open 
> source version that must have (fix) commit abcd in its commit history"  
> Then, once our fix is upstreamed, the open source version will become a 
> candidate.  I suspect that's too much to ask though!
>
>  
>
>>
>>
>> However, what I can express to the tool is
>>>
>>> My module requires version ??? of dependency x/y/z
>>>
>>> Replace  x/y/z @ version ??? with  @ abcd1234
>>>
>>>  
>>>
>>>
>> Currently it should be `v0.0.0-0001010100-`.
>> We're also rethinking this behavior for `replace` in general; see 
>> https://golang.org/issue/33370.
>>
>>
>
> Good to know that there's a good value to put there.  Would be nice if it 
> could be made shorter, though, I'll have to look that up.  
>
>
>>>- There's nowhere to specify the details of the repo (e.g. 
>>>connection/auth type), all that has to magically work according to my 
>>> git 
>>>settings and the defaults aren't great for private repos (which we 
>>> access 
>>>over ssh).
>>>
>>> See https://golang.org/cmd/go/#hdr-Remote_import_paths.
>>
>> In general we expect that either you have an HTTPS server to locate the 
>> repo for a given import path (credentials can be stored in a `.netrc` file; 
>> see also https://golang.org/issue/26232), or you include an explicit VCS 
>> extension somewhere in the import path (and have corresponding credentials 
>> configured per-user in your VCS).
>>
>
>
> Maybe I'm just out-of-date, go now tries git+ssh as well as https://, was 
> that new in v1.13?  Since we only use github, that should be good enough 
> for me.
>
>
>
>>
>> We also just ran into the new GOPRIVATE env var and had to update all our 
>>> builds to use that.  Couldn't that just fall back to the private behaviour 
>>> if it gets a cache miss, it seems over-engineered to require an explicit 
>>> exception!
>>>
>>
>> The checksum database and proxy fallback are designed to provide 
>> integrity by default.
>>
>> Without making the opt-out explicit, an origin could serve one set of 
>> sources (and checksum) to y

[go-nuts] Re: Struggling with go mod, micro-repos and private forks

[Shaun Crampton]

>
>
> The good news is that we're aware of (and planning to address) most of 
> these pain points; the bad news is that we haven't been able to get to most 
> of them yet.
>
>
Great to see, thanks!

 

>
>> I think the biggest problem we have is when working with private repos.  
>>  What I want to express to the tool is 
>>
>> My module requires commit abcd1234 (or version v1.2.3) of dependency x/y/z
>>
>> Look for any instances of dependency x/y/z in git repo 
>> g...@github.com:ourfork instead of the default.
>>
>>  
> I believe this is https://golang.org/issue/28176.
>
> It's an intriguing idea, but problematic if the dependencies are specified 
> as pseudo-versions, because the commit hashes in a fork in general will not 
> match the hashes in the upstream module. We're still exploring alternatives 
> for this use-case.
>


So, the problem that pseudo-versions are trying to solve is ordering in the 
face of unordered commit hashes?  For my use case, the way I want this to 
work is that, once a dependency is pinned to my private fork, it's as if 
the public repo didn't exist.  In 99% of cases, all the pins that I'm 
working with are my organisation's repos anyway so once we pin to a private 
fork that's what we want to use everywhere, transitively, even in 
downstream repos.  In extreme cases, even the version numbering scheme in 
the private fork may be different so there's not necessarily any relation 
between the open source v2.3.0 and the private v2.3.0 tags.

Our minority use case is forking some public repo to make a fix.  In that 
case, a perfect tool would have a way to say "we need a version >= open 
source version that must have (fix) commit abcd in its commit history"  
Then, once our fix is upstreamed, the open source version will become a 
candidate.  I suspect that's too much to ask though!

 

>
>
> However, what I can express to the tool is
>>
>> My module requires version ??? of dependency x/y/z
>>
>> Replace  x/y/z @ version ??? with  @ abcd1234
>>
>>  
>>
>>
> Currently it should be `v0.0.0-0001010100-`.
> We're also rethinking this behavior for `replace` in general; see 
> https://golang.org/issue/33370.
>
>

Good to know that there's a good value to put there.  Would be nice if it 
could be made shorter, though, I'll have to look that up.  


>>- There's nowhere to specify the details of the repo (e.g. 
>>connection/auth type), all that has to magically work according to my git 
>>settings and the defaults aren't great for private repos (which we access 
>>over ssh).
>>
>> See https://golang.org/cmd/go/#hdr-Remote_import_paths.
>
> In general we expect that either you have an HTTPS server to locate the 
> repo for a given import path (credentials can be stored in a `.netrc` file; 
> see also https://golang.org/issue/26232), or you include an explicit VCS 
> extension somewhere in the import path (and have corresponding credentials 
> configured per-user in your VCS).
>


Maybe I'm just out-of-date, go now tries git+ssh as well as https://, was 
that new in v1.13?  Since we only use github, that should be good enough 
for me.



>
> We also just ran into the new GOPRIVATE env var and had to update all our 
>> builds to use that.  Couldn't that just fall back to the private behaviour 
>> if it gets a cache miss, it seems over-engineered to require an explicit 
>> exception!
>>
>
> The checksum database and proxy fallback are designed to provide integrity 
> by default.
>
> Without making the opt-out explicit, an origin could serve one set of 
> sources (and checksum) to you, and another version to someone else, and 
> simply refuse to serve anything at all in response to queries from the 
> checksum database (so that the entry would always be missing), and no one 
> would be the wiser.
>


Fair enough, I suppose.  From a selfish point of view, making the go 
tooling auto-trust github.com private repos would be nice :-)  I can't 
easily do that with a single entry in GOPRIVATE since whether a repo is 
private or not can only be guessed by whether you need to use credentials 
or ssh to access it.

 

>
>
> Another issue we've had is making a build mode where we can build against 
>> local copies of the dependencies for quick development.  We can add replace 
>> directives to point to local directories, which is part of the puzzle, but 
>> it's hard to do that "just for one build".  Not sure what we're looking for 
>> here; maybe an optional go.mod override file that can be passed in for just 
>> one build?
>>
>
> You mean like https://golang.org/issue/34506? 🙂 
>
>  
>

Sounds like just what we need, thanks! 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/bbab7527-1c46-40dd-ad4a-b48c7a8c793

[go-nuts] Struggling with go mod, micro-repos and private forks

[Shaun Crampton]

Hi All,

My team own a large product that sprawls across a lot of repos.  We also 
manage a private, commercial fork and we occasionally do new development in 
private and then merge across to open source.  While a monorepo might have 
been easier to manage, we're stuck with what we've got.  We've recently 
moved to go mod and it just seems like we're constantly fighting the tool.  
I'm hoping you can either suggest some good workflows or maybe improve the 
tool!

I think the biggest problem we have is when working with private repos.  
 What I want to express to the tool is 

My module requires commit abcd1234 (or version v1.2.3) of dependency x/y/z

Look for any instances of dependency x/y/z in git repo 
g...@github.com:ourfork instead of the default.


However, what I can express to the tool is

My module requires version ??? of dependency x/y/z

Replace  x/y/z @ version ??? with  @ abcd1234

 

This throws up a couple of problems:

   - What should version ??? be?  It's only there to be replaced, which 
   seems like a bit of a smell.  
  - If I set it to the commit ID it gets resolved and I have to change 
  three places in the file each time I move the pin.
  - If I set it to a particular version, that seems misleading.  
  - I guess I can set it to v0.0.0, but again that seems misleading.
   - There's nowhere to specify the details of the repo (e.g. 
   connection/auth type), all that has to magically work according to my git 
   settings and the defaults aren't great for private repos (which we access 
   over ssh).

We also just ran into the new GOPRIVATE env var and had to update all our 
builds to use that.  Couldn't that just fall back to the private behaviour 
if it gets a cache miss, it seems over-engineered to require an explicit 
exception!

Another issue we've had is making a build mode where we can build against 
local copies of the dependencies for quick development.  We can add replace 
directives to point to local directories, which is part of the puzzle, but 
it's hard to do that "just for one build".  Not sure what we're looking for 
here; maybe an optional go.mod override file that can be passed in for just 
one build?  

-Shaun

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/208c32de-2c61-4148-9c31-3e11d4cebbd4%40googlegroups.com.

[go-nuts] go install seems to do a lot of rebuilding in a container

[Shaun Crampton]

Hi,

I'm trying to optimize the build of my app.  Our build times have crept up 
to ~60s as we've added more dependencies and it's starting to become a 
problem.

The primary problem seems to be that we're building in a container, so 
we're losing the $GOPATH/pkg cache of build artefacts between builds. 
 However, even if I bind the same directory to that location for each run, 
it doesn't seem to help.  The artefacts (.a files) get built and added to 
that directory but the following build seems to ignore them and rebuild 
them anyway.

If I do two builds in the same container then it does pick up the 
artefacts.  Likewise, if I build outside the container it seems to work as 
expected.

I feel like there's something I'm missing.  Please can you help me 
understand how "go install" determines the freshness of artefacts and why 
they might not be being used? I tried running with -x but I can't see any 
info about freshness (but then I'm not sure what I'm looking for in that 
output).

Thanks,

-Shaun

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Function pointer vs interface for callback?

[Shaun Crampton]

Thanks for the tips.

On Wed, Oct 12, 2016 at 2:11 PM Jesse McNelis  wrote:

> On Wed, Oct 12, 2016 at 11:05 PM,   wrote:
> >  Seems like a function
> > pointer is more universal (I don't even need an object to be a receiver)
> but
> > maybe an interface is more idiomatic?
>
> An interesting pattern from the http pkg is that a function type can
> also implement an interface.
> https://golang.org/pkg/net/http/#HandlerFunc
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.