[go-nuts] Re: SIGILL running 1.14 on macOS

[James Pettyjohn]

Sooner the better, it's crept into downstream packages. E.g. docker 
installed via brew is toast as it was built in 1.14.

On Wednesday, March 4, 2020 at 1:16:13 PM UTC-8, Jon Conradt wrote:
>
> You can follow the progress toward 1.14.1 via 
> https://github.com/golang/go/milestone/137
>
> Looking at 1.13.0 and 1.12.0 it looks like the time between a .0 release 
> and a .0 release is about three weeks. So I'm hoping for a St. Patrick's 
> Day releases of 1.14.1
>
> Jon
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e30f0353-e0df-4970-b319-9cc2e8e595ff%40googlegroups.com.

[go-nuts] Re: SIGILL running 1.14 on macOS

[James Pettyjohn]

Looks like this was already resolved for 1.14.1 
<https://github.com/golang/go/issues/37459>.

On Wednesday, February 26, 2020 at 11:00:24 AM UTC-8, James Pettyjohn wrote:
>
> A little further digging shows that VZEROUPPER is part of the AVX 
> instruction set, not supported on any Apple computers prior to 2011 from 
> what I can tell.
>
> Given the rc1 code worked fine, that could be an alternative for Darwin 
> where the CPU does not support AVX. While not ideal, I'd argue it's better 
> than cutting off support for these machines which can run High Sierra. Once 
> support for High Sierra is dropped then it should not matter as it takes a 
> 2013 or later model to run anything after High Sierra.
>
> - James
>
>
>
> On Wednesday, February 26, 2020 at 10:29:33 AM UTC-8, James Pettyjohn 
> wrote:
>>
>> I am trying to run go1.14 on macOS 10.13.2 with on a late 2009 iMac with 
>> an i7 but always get the same crash:
>>
>>
>> SIGILL: illegal instruction
>> PC=0x1066040 m=0 sigcode=1
>>
>>
>> goroutine 1 [running, locked to thread]:
>> runtime.asyncPreempt()
>> /usr/local/go/src/runtime/preempt_amd64.s:8 fp=0xc58ef0 sp=
>> 0xc58ee8 pc=0x1066040
>> fmt.init()
>> /usr/local/go/src/fmt/scan.go:465 fp=0xc58ef8 sp=0xc58ef0 
>> pc=0x10e6540
>> runtime.doInit(0x1a80520)
>> /usr/local/go/src/runtime/proc.go:5414 +0x8a fp=0xc58f28 sp=
>> 0xc58ef8 pc=0x1043aca
>> runtime.doInit(0x1a80ec0)
>> /usr/local/go/src/runtime/proc.go:5409 +0x57 fp=0xc58f58 sp=
>> 0xc58f28 pc=0x1043a97
>> runtime.doInit(0x1a850c0)
>> /usr/local/go/src/runtime/proc.go:5409 +0x57 fp=0xc58f88 sp=
>> 0xc58f58 pc=0x1043a97
>> runtime.main()
>> /usr/local/go/src/runtime/proc.go:190 +0x1ce fp=0xc58fe0 sp=
>> 0xc58f88 pc=0x103702e
>> runtime.goexit()
>> /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc58fe8 
>> sp=0xc58fe0 pc=0x10647d1
>>
>>
>> rax0x1a80520
>> rbx0xb
>> rcx0x0
>> rdx0x1a80578
>> rdi0xcb2020
>> rsi0x10e6540
>> rbp0xc58f18
>> rsp0xc58ee8
>> r8 0x3277901
>> r9 0x203000
>> r100x8
>> r110x55
>> r120x1
>> r130x55
>> r140x171acac
>> r150x38
>> rip0x1066040
>> rflags 0x10202
>> cs 0x2b
>> fs 0x0
>> gs 0x0
>>
>>
>> Which appears to be a call to VZEROUPPER in preempt_amd64.s 
>> <https://github.com/golang/go/blob/go1.14/src/runtime/preempt_amd64.s#L8>
>> .
>>
>> This was not the case in rc1, which runs just fine. I get the same 
>> results from installing it with brew, .pkg and .tgz installs - have done 
>> the install from source yet.
>>
>> Any tips would be appreciated.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ebfc97e6-bf43-46cc-bfbd-cbe4b756353d%40googlegroups.com.

[go-nuts] Re: SIGILL running 1.14 on macOS

[James Pettyjohn]

A little further digging shows that VZEROUPPER is part of the AVX 
instruction set, not supported on any Apple computers prior to 2011 from 
what I can tell.

Given the rc1 code worked fine, that could be an alternative for Darwin 
where the CPU does not support AVX. While not ideal, I'd argue it's better 
than cutting off support for these machines which can run High Sierra. Once 
support for High Sierra is dropped then it should not matter as it takes a 
2013 or later model to run anything after High Sierra.

- James



On Wednesday, February 26, 2020 at 10:29:33 AM UTC-8, James Pettyjohn wrote:
>
> I am trying to run go1.14 on macOS 10.13.2 with on a late 2009 iMac with 
> an i7 but always get the same crash:
>
>
> SIGILL: illegal instruction
> PC=0x1066040 m=0 sigcode=1
>
>
> goroutine 1 [running, locked to thread]:
> runtime.asyncPreempt()
> /usr/local/go/src/runtime/preempt_amd64.s:8 fp=0xc58ef0 sp=
> 0xc58ee8 pc=0x1066040
> fmt.init()
> /usr/local/go/src/fmt/scan.go:465 fp=0xc58ef8 sp=0xc58ef0 
> pc=0x10e6540
> runtime.doInit(0x1a80520)
> /usr/local/go/src/runtime/proc.go:5414 +0x8a fp=0xc58f28 sp=
> 0xc58ef8 pc=0x1043aca
> runtime.doInit(0x1a80ec0)
> /usr/local/go/src/runtime/proc.go:5409 +0x57 fp=0xc58f58 sp=
> 0xc58f28 pc=0x1043a97
> runtime.doInit(0x1a850c0)
> /usr/local/go/src/runtime/proc.go:5409 +0x57 fp=0xc58f88 sp=
> 0xc58f58 pc=0x1043a97
> runtime.main()
> /usr/local/go/src/runtime/proc.go:190 +0x1ce fp=0xc58fe0 sp=
> 0xc58f88 pc=0x103702e
> runtime.goexit()
> /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc58fe8 sp
> =0xc58fe0 pc=0x10647d1
>
>
> rax0x1a80520
> rbx0xb
> rcx0x0
> rdx0x1a80578
> rdi0xcb2020
> rsi0x10e6540
> rbp0xc58f18
> rsp0xc58ee8
> r8 0x3277901
> r9 0x203000
> r100x8
> r110x55
> r120x1
> r130x55
> r140x171acac
> r150x38
> rip0x1066040
> rflags 0x10202
> cs 0x2b
> fs 0x0
> gs 0x0
>
>
> Which appears to be a call to VZEROUPPER in preempt_amd64.s 
> <https://github.com/golang/go/blob/go1.14/src/runtime/preempt_amd64.s#L8>.
>
> This was not the case in rc1, which runs just fine. I get the same results 
> from installing it with brew, .pkg and .tgz installs - have done the 
> install from source yet.
>
> Any tips would be appreciated.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3ca625a5-c45e-4f05-92f9-36e3e61ff739%40googlegroups.com.

[go-nuts] SIGILL running 1.14 on macOS

[James Pettyjohn]

I am trying to run go1.14 on macOS 10.13.2 with on a late 2009 iMac with an 
i7 but always get the same crash:


SIGILL: illegal instruction
PC=0x1066040 m=0 sigcode=1


goroutine 1 [running, locked to thread]:
runtime.asyncPreempt()
/usr/local/go/src/runtime/preempt_amd64.s:8 fp=0xc58ef0 sp=
0xc58ee8 pc=0x1066040
fmt.init()
/usr/local/go/src/fmt/scan.go:465 fp=0xc58ef8 sp=0xc58ef0 pc
=0x10e6540
runtime.doInit(0x1a80520)
/usr/local/go/src/runtime/proc.go:5414 +0x8a fp=0xc58f28 sp=
0xc58ef8 pc=0x1043aca
runtime.doInit(0x1a80ec0)
/usr/local/go/src/runtime/proc.go:5409 +0x57 fp=0xc58f58 sp=
0xc58f28 pc=0x1043a97
runtime.doInit(0x1a850c0)
/usr/local/go/src/runtime/proc.go:5409 +0x57 fp=0xc58f88 sp=
0xc58f58 pc=0x1043a97
runtime.main()
/usr/local/go/src/runtime/proc.go:190 +0x1ce fp=0xc58fe0 sp=
0xc58f88 pc=0x103702e
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc58fe8 sp=
0xc58fe0 pc=0x10647d1


rax0x1a80520
rbx0xb
rcx0x0
rdx0x1a80578
rdi0xcb2020
rsi0x10e6540
rbp0xc58f18
rsp0xc58ee8
r8 0x3277901
r9 0x203000
r100x8
r110x55
r120x1
r130x55
r140x171acac
r150x38
rip0x1066040
rflags 0x10202
cs 0x2b
fs 0x0
gs 0x0


Which appears to be a call to VZEROUPPER in preempt_amd64.s 
.

This was not the case in rc1, which runs just fine. I get the same results 
from installing it with brew, .pkg and .tgz installs - have done the 
install from source yet.

Any tips would be appreciated.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5fa8d757-9f5a-474c-9ae1-79deaff08d0a%40googlegroups.com.

[go-nuts] Marshaling structs to command line arguments

[James Pettyjohn]

Hi,

While not the most likely of scenarios, I'm having to write a lot of 
integration around existing command line tools and want a better way then 
passing strings/constants around.

While there are ample choices to go from command line arguments to structs 
etc, I'm not finding anything besides vanilla os/exec for command line 
construction.

Most encoding packagers go from a struct or the like and generate marshal 
the serialized item. Anyone seen this same thing but making an array of 
args consumable by exec?

- J

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6b1e926b-2f45-4b5f-bc50-a49da41ae74f%40googlegroups.com.

Re: [go-nuts] Parsing go.mod

[James Pettyjohn]

Thanks a lot for the tips. This repo is working out and should hold me 
over, as Ian mentioned, until golang.org/x/mod/modfile is out.

On Tuesday, September 3, 2019 at 12:26:28 AM UTC-7, Paul Jolly wrote:
>
> Also note that you can run: 
>
> go mod edit -json 
>
> and get JSON output. 
>
> go help mod edit 
>
> will given you the types you can then use to unmarshal the JSON. 
>
> Failing that, the code that Dan referenced has been factored out into: 
>
> https://godoc.org/github.com/rogpeppe/go-internal/modfile 
>
> On Tue, 3 Sep 2019 at 08:21, James Pettyjohn  > wrote: 
> > 
> > Thanks. I saw the same being done by vgo - might be viable. 
> > 
> > On Monday, September 2, 2019 at 10:52:45 PM UTC-7, kortschak wrote: 
> >> 
> >> Not really exposed, but there is code you could copy. 
> >> 
> >> https://golang.org/pkg/cmd/go/internal/modfile/#Parse 
> >> 
> >> On Mon, 2019-09-02 at 22:44 -0700, James Pettyjohn wrote: 
> >> > Hi, 
> >> > 
> >> > This might be a bad idea but I'm trying to parse the go.mod file for 
> >> > data 
> >> > as part of my build process - if at all possible I'd like to avoid 
> >> > duplicating the data that is already there in another file. 
> >> > 
> >> > Is there an exposed package that can be used for this? 
> >> > 
> >> > - J 
> >> > 
> >> 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to golan...@googlegroups.com . 
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/7a06f505-187e-473f-b748-a8bf7f114bf6%40googlegroups.com.
>  
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2a4fd111-ffaf-4c9c-a427-f10c04af6f35%40googlegroups.com.

Re: [go-nuts] Parsing go.mod

[James Pettyjohn]

That's good news, gives me some confidence that the tooling will keep 
working after 1.14.

On Tuesday, September 3, 2019 at 6:43:15 PM UTC-7, Ian Cottrell wrote:
>
> The parsing code will eventually be public, see 
> https://github.com/golang/go/issues/31761
> When it is done, you would want golang.org/x/mod/modfile, which will be a 
> (possibly modified) copy of the compiler internal package of the same name.
>
>
> On Tue, Sep 3, 2019 at 3:29 AM Paul Jolly > 
> wrote:
>
>> > Sorry I left out the details. So far it's two specific properties I 
>> need for instructions in my build process:
>> >
>> > 1) The full name of the package
>>
>> go list -f {{.Name}} example.com/blah
>>
>> > 2) The version of the protobuf dependency for locking the right version 
>> protoc-gen-go before running protoc (that has been asked quite a few times, 
>> total hack, but works).
>>
>> go list -f {{.Version}} -m example.com/blah
>>
>> Note that if you have a replace directive for example.com/blah then
>> the replace target might have a version.
>>
>> For more detail on either of the above see:
>>
>> go help list
>>
>> >
>> > Best,
>> > James
>> >
>> > On Monday, September 2, 2019 at 10:58:53 PM UTC-7, Kurtis Rader wrote:
>> >>
>> >> On Mon, Sep 2, 2019 at 10:44 PM James Pettyjohn  
>> wrote:
>> >>>
>> >>> This might be a bad idea but I'm trying to parse the go.mod file for 
>> data as part of my build process - if at all possible I'd like to avoid 
>> duplicating the data that is already there in another file.
>> >>
>> >>
>> >> In this type of situation you should explain why you need to parse a 
>> go.mod file. Especially since you seem to be asking something not already 
>> asked a thousand times. It is possible, perhaps likely, the reason you want 
>> to do this can be solved some other way.
>> >>
>> >> --
>> >> Kurtis Rader
>> >> Caretaker of the exceptional canines Junior and Hank
>> >
>> > --
>> > You received this message because you are subscribed to the Google 
>> Groups "golang-nuts" group.
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> an email to golan...@googlegroups.com .
>> > To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/8b716cac-0eb6-4e95-bdde-12ee832e06a4%40googlegroups.com
>> .
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golan...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CACoUkn4MouSb-spFS-NvdYZdC827%2Bx28pXLD0ABd%3D52AHBvfaw%40mail.gmail.com
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/23c433db-83b1-4fb9-b8c6-99631b0f3952%40googlegroups.com.

Re: [go-nuts] Parsing go.mod

[James Pettyjohn]

Sorry I left out the details. So far it's two specific properties I need 
for instructions in my build process:

1) The full name of the package
2) The version of the protobuf dependency for locking the right 
version protoc-gen-go before running protoc (that has been asked quite a 
few times, total hack, but works).

Best,
James

On Monday, September 2, 2019 at 10:58:53 PM UTC-7, Kurtis Rader wrote:
>
> On Mon, Sep 2, 2019 at 10:44 PM James Pettyjohn  > wrote:
>
>> This might be a bad idea but I'm trying to parse the go.mod file for data 
>> as part of my build process - if at all possible I'd like to avoid 
>> duplicating the data that is already there in another file.
>>
>
> In this type of situation you should explain why you need to parse a 
> go.mod file. Especially since you seem to be asking something not already 
> asked a thousand times. It is possible, perhaps likely, the reason you want 
> to do this can be solved some other way.
>  
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8b716cac-0eb6-4e95-bdde-12ee832e06a4%40googlegroups.com.

Re: [go-nuts] Parsing go.mod

[James Pettyjohn]

Thanks. I saw the same being done by vgo - might be viable.

On Monday, September 2, 2019 at 10:52:45 PM UTC-7, kortschak wrote:
>
> Not really exposed, but there is code you could copy. 
>
> https://golang.org/pkg/cmd/go/internal/modfile/#Parse 
>
> On Mon, 2019-09-02 at 22:44 -0700, James Pettyjohn wrote: 
> > Hi, 
> > 
> > This might be a bad idea but I'm trying to parse the go.mod file for 
> > data 
> > as part of my build process - if at all possible I'd like to avoid 
> > duplicating the data that is already there in another file. 
> > 
> > Is there an exposed package that can be used for this? 
> > 
> > - J 
> > 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7a06f505-187e-473f-b748-a8bf7f114bf6%40googlegroups.com.

[go-nuts] Parsing go.mod

[James Pettyjohn]

Hi,

This might be a bad idea but I'm trying to parse the go.mod file for data 
as part of my build process - if at all possible I'd like to avoid 
duplicating the data that is already there in another file.

Is there an exposed package that can be used for this?

- J

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/79c5a541-9f66-4907-9a58-8a74c7fb349c%40googlegroups.com.

[go-nuts] Go as your first language

[James Pettyjohn]

I've had multiple occasions where I've needed to train someone to be a 
programmer from scratch in a Go environment.

Trouble I've found is while the go texts are simple and straightforward, 
relatively speaking, they often written by someone who sought a better life 
in go, fleeing Java/C/C++. They will routinely reference these other 
languages in examples, touting the benefits of go is comparison to the old 
language. Much like reading GOF design patterns without a background in 
smalltalk, it is hard for new developers to pick up when they don't know 
other languages first. Commonly they cut it back and learn JS first.

Assuming they eventually picked up the language they now need to learn how 
to be a software engineer and write code that doesn't suck. Especially 
present with those who just learned how to program using JS. And what I've 
seen on the subject often expects a knowledge of another language.

Are there tracks of knowledge to take someone from 0 to understanding 
baseline knowledge?  

And from there through taking them to a professional grade standard?

Best,
James

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Overriding cgo version

[James Pettyjohn]

Just realized the issue here is this was assumed to be disabled when build 
the package because of the build tag "icu". This conflicts with the 
dependency of icu for production code in collate/tools/colcmp/icu.go. 
Either use an alternative build tag or take out the hard version dependency 
(making it then consistent with collate).

Best,
James

On Wednesday, April 26, 2017 at 5:06:57 PM UTC-7, Ian Lance Taylor wrote:
>
> [ + mpvl ] 
>
> On Wed, Apr 26, 2017 at 4:39 PM, James Pettyjohn <japet...@gmail.com 
> > wrote: 
> > I just updated my dependency on golang.org/x/text and found they added 
> a 
> > hard depedency on ICU 57. 
> > 
> >> ld: library not found for -licui18n.57 
> >> clang: error: linker command failed with exit code 1 (use -v to see 
> >> invocation) 
> > 
> > 
> > Trying to build this on a mac where homebrew installed icu4c as it 
> brings in 
> > icu 58. 
> > 
> > This appears to be caused by 
> > https://github.com/golang/text/blob/master/cases/icu.go#L16 . 
> > 
> > Is there a way to override this at compilation time? 
>
> No.  It's just a test, though.  Perhaps there needs to be a simple way 
> to disable, or to enable, that test. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: KVs store for large strings

[James Pettyjohn]

Thanks for the follow-up!

That ended up being the approach I took - map two levels deep (because 
that's how the keys come through) and load everything at start up and use 
as an immutable structure. While in dev it is using the old code 
essentially and that's fine.

I also noted that internal structure in the old code using a struct {x 
string, y string} for its first level key which I thought odd. Not sure 
what effects that would have on things.

Best,
James

On Thursday, June 15, 2017 at 8:17:47 AM UTC-7, ksug wrote:
>
> There were mistakes with the code I posted. Oh well, in a hurry and not 
> the point. 
>
>
>
> On 15/06/17 22:41, Kiki Sugiaman wrote: 
> > I think I misunderstood what you were trying to accomplish. Let me try 
> > to rephrase and see if I'm getting closer: 
> > 
> > - to accommodate template changes during development, you introduced 
> > read-locking to your app. 
> > - as a result, your reads in production are slow. Never mind the fact 
> > that locking is pointless in production because the templates don't 
> change. 
> > - ideally, you want your reads to yield to template changes during 
> > development, but go lockless in production. 
> > 
> > If the above are correct, then the solution is not data structure 
> > change, but code swapping during init() depending on the environment 
> > variable $DEVMODE (or whatever you want to call it). 
> > 
> > Example: 
> > https://play.golang.org/p/wqb9lVIPLa 
> > (repeat several times and change the random seed to randomize the 
> > behavior a bit) 
> > 
> > Of course, optimizing the data structure doesn't hurt. And to answer 
> > your question, yes map can handle 10's of thousands entries with 100MB 
> > raw data with *relative* ease. 
> > 
> > 
> > 
> > On 15/06/17 15:45, James Pettyjohn wrote: 
> >> I looked at doing boltdb but ended up with a slightly simpler 
> >> approach. I don't need the disk persistence in this case. 
> >> 
> >> I tried radix tree implementation but found it slower than my current 
> >> nested map with locks. In that interest I rewrote the nested map 
> >> approach considering the key partitioning noted by Robert. 
> >> 
> >> The primary differences in my approach this time were: 
> >> 1) Read the whole dataset once treat it as immutable, no locking 
> >> 2) Align the API to no require any manipulation of the keys after the 
> >> first load, so every lookup is now 0 allocation. 
> >> 
> >> The structure is a simplistic map[string]map[string]string - language 
> >> -> key. There was a breakdown of the second map further in the 
> >> previous implementation, probably bringing it from a few thousand to a 
> >> few hundred each. Don't know what the threshold on the benefits there, 
> >> based on my experience the map implementation can handle 10s of 
> >> thousands with ease. 
> >> 
> >> This brought the lookups down by about 50%: 
> >> 
> >> BenchmarkLookupLooseLazy-8 300 458 ns/op 32 B/op 1 allocs/op 
> >> BenchmarkLookupLooseMap-8 500 265 ns/op 0 B/op 0 allocs/op 
> >> 
> >> Let me know if see anything else worth adjusting. 
> >> 
> >> Best, 
> >> James 
> >> 
> >> 
> >> On Wednesday, June 14, 2017 at 2:01:44 PM UTC-7, Robert Johnstone 
> wrote: 
> >> 
> >> I'm surprised that the memory overhead is significant - 100 MB is 
> >> not that much. 
> >> 
> >> Assuming that you don't need atomic updates to the entire KV store, 
> >> partition the keys. 
> >> 
> >> Does the periodic reload involve changing the keys?  If not, you 
> >> could map the dataset into nested structs.  However, you will still 
> >> need to synchronise access if you want to to reload without 
> stopping 
> >> the server, but that would just be the leaves.  Switching just the 
> >> top-tier to a struct could help with the contention. 
> >> 
> >> 
> >> On Wednesday, 14 June 2017 14:45:25 UTC-4, James Pettyjohn wrote: 
> >> 
> >> I have an application which has all of it's text, multiple 
> >> languages, stored in XML on disk that is merged into templates 
> >> on the fly. About 100MB. Templates use dozens of strings for 
> >> each render. 
> >> 
> >> Currently this is loaded in full into memory in a bunch of tier 
> >> hash maps. They are lazy loaded and using multiple locks to 
> 

[go-nuts] KVs store for large strings

[James Pettyjohn]

I have an application which has all of it's text, multiple languages, 
stored in XML on disk that is merged into templates on the fly. About 
100MB. Templates use dozens of strings for each render. 

Currently this is loaded in full into memory in a bunch of tier hash maps. 
They are lazy loaded and using multiple locks to perform reads but, unless 
in dev mode, actually don't change throughout the lifetime of the 
application and should be considered immutable.

While workable at a smaller scale, it's slow at scale. The most important 
factor is concurrent lookup speed, secondary concern is memory overhead. 
And it cannot preclude periodic reload while doing dev.

Is there a data structure or lib that suits this scenarios more than others?

Best,
James

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go install speedup within docker

[James Pettyjohn]

Hi Vlad,

I am using go install and the pkg directory is being mapped into the 
container.

docker run -ti --rm \
 -v "$PWD":/usr/src/pjt \ # Mount the entire project to this directory
 -w /usr/src/pjt \ # use the project dir as the cwd
 -e "GOPATH=/usr/src/pjt" \ # set the GOPATH the base of the project dir
 -e "STATICBUILD=1" \
 -e "PREBUILD_DEPS=$PREBUILD_DEPS" \
gobuilder:1.8-alpine \
 ./build.sh

Then in build.sh (naming is misleading and ambiguous as it does not use go 
build, but instead go install) does a couple of things:

# set the go path to he current dir
export GOPATH=`pwd` 

# install the project, give a suffix in the pkg directory for docker built 
files
go install -installsuffix docker \
-v --tags netgo --ldflags '-extldflags "-lm -lstdc++ -static"' \
org/project 

Looks like I figured it out though:

1) "-installsuffix" effects packages in GOPATH and *GOROOT* - so all of the 
standard libs were being built newly and GOROOT was in the docker image 
only, so that was rebuilt every time - and I can only assume this caused 
everything else to rebuild due to dependencies. I removed this and cut it 
down from 1 1/2 minutes to 19 seconds. This is not needed in my case as it 
turns out.

2) The build time for no changes should be closed to no time at all, a 
couple seconds from my tests. This turned out to be "--tags netgo" which 
looks like it caused a rebuild of the "net" package and the whole tree on 
top of that. I was able to remove that and shave the rebuild time down to 3 
seconds including docker spin-up.

netgo is probably a needed item in other static builds. In the my case it 
wasn't necessary. Building from alpine docker and then making the resultant 
binary to run as it's own base image, e.g. dockerfile:

FROM scratch
COPY bin/pjt /bin/pjt
ENTRYPOINT ["/bin/pjt"]

Turning off netgo increased the build by 6 megs, but 43 over 37MB is worth 
the tradeoff for quick rebuilds. And for the use case where it's using 
docker - it still works.

This procedure is touched on throughout the net, hopefully down the road it 
is more codified.

Best,
James 

On Monday, June 12, 2017 at 3:06:49 AM UTC-7, Vladimir Varankin wrote:
>
> I think the difference is that I use my project root as my GOPATH, so pkg 
> directory ($GOPATH/pkg), which stores the build cache, is mounted inside 
> the container every time I run the build. What if you try
>
> ```
> docker container run --rm -v $PWD:/usr/src/prj -v $PWD/_build:/usr/pkg ...
> ```
>
> Doing so, should keep pkg caches between builds. Not sure if it still 
> valid, but you might consider to switch `go install` (see 
> https://dave.cheney.net/2014/06/04/what-does-go-build-build)
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go install speedup within docker

[James Pettyjohn]

Hey Vlad,

Here's the docker command:

docker run -ti --rm \
 -v "$PWD":/usr/src/pjt \
 -w /usr/src/pjt \
 -e "GOPATH=/usr/src/pjt" -e "STATICBUILD=1" \
 -e "PREBUILD_DEPS=$PREBUILD_DEPS" \
gobuilder:1.8-alpine \
 ./build.sh

Regarding my docker:
docker-machine version 0.10.0, build 76ed2a6
Client:
 Version:  17.03.0-ce
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   60ccb22
 Built:Thu Mar  2 01:11:00 2017
 OS/Arch:  darwin/amd64

Server:
 Version:  17.03.0-ce
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   3a232c8
 Built:Tue Feb 28 07:52:04 2017
 OS/Arch:  linux/amd64
 Experimental: false

Running on macOS.

Best,
James

On Sunday, June 11, 2017 at 7:00:18 AM UTC-7, Vladimir Varankin wrote:
>
> Hey James, 
>
> Could you show the docker run command, which you invoke to enter a 
> container? 
>
> I use docker to build my application as well, so I just do `docker 
> container run --rm -ti --volume $PWD:/gocode/src/app --workdir 
> /gocode/src/app  go build`. Doing so with container 
> reusage or not, I haven't found any speed difference so far.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Go install speedup within docker

[James Pettyjohn]

I have a statically built go binary with cgo deps (wellington/go-libsass) 
which I'm trying to speed up the rebuild time for.

This is run within a docker image because so the statically compiled binary 
will run as it's own base image on docker.

The docker image the build is run within is:
FROM golang:1.8-alpine


# We need g++ for libraries like libsass
RUN apk --no-cache add g++ icu-libs make

This is the build command for the project:
go install -installsuffix docker \
-v --tags netgo --ldflags '-extldflags "-lm -lstdc++ -static"' \
org/project

Running this within the same docker instance first speeds up the build:

go install -installsuffix docker \
-v --tags netgo --ldflags '-extldflags "-lm -lstdc++ -static"' \
github.com/wellington/go-libsass


But running that doesn't speed up the overall build process as  the next 
time I run a new docker container, same folder on the host, and try to 
build the project without changing any files then it takes the full time 
again.

What could cause go install to disregard this separate build done?

Best,
James

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: In-memory secondary index

[James Pettyjohn]

Thanks Dragos, I had looked at that one earlier but I was hoping for 
something that was more implicit to a struct definition.

Something like:

type StructIndex struct{}
func NewStructIndex([]Interface{}) *StructIndex{}
func (si *StructIndex) Get(k string, v interface{}) []interface{}

Implementation would probably include an Index(fieldName) call as well or 
it could create one based on heuristics.

I could put that together using maps or an radix-tree library, but I know I 
would not take advantage of everything I library on the matter would.

In the mean time I'm trying widely different approach that may fit in my 
scenario with cayley. 

On Monday, June 5, 2017 at 4:37:59 PM UTC-7, Dragos Harabor wrote:
>
> Something like this?
> https://github.com/hashicorp/go-memdb
>
> Though it may have more features than you need. Or features that you don't 
> yet know you want :-)
>
> On Monday, June 5, 2017 at 3:28:33 AM UTC-7, James Pettyjohn wrote:
>>
>> I'm loading a set of structs in memory and will be frequently using them 
>> throughout the lifetime of the application and doing frequent 'queries' 
>> against the set to get those where certain fields have certain values.
>>
>> Are there are any native go libraries which do this kind of sort of 
>> "secondary index" on in memory data structures?
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Golang http reverse proxy in production

[James Pettyjohn]

I'm looking at using the stock HTTP reverse proxy, briefly looking at the 
implementation it seems to be ready to withstand a production workload.

Any cautions or caveats in going this route?

- J

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Project architecture

[James Pettyjohn]

I tried this once and I don't know that I effectively decoupled the direct 
cgo dependency. Any pointers on effectively doing this?

On Saturday, March 11, 2017 at 11:22:03 PM UTC-8, Tamás Gulácsi wrote:
>
> Factor out the cgo related part into a wrapper package (subdir), that will 
> help with the compile times.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Project architecture

[James Pettyjohn]

Aye, this is some ways down the road now - quite a few projects on this 
base template. Only real complaint is:

- Submodules get out of date where data is concerned
- Some submodules contain a lot of data and so bloat deployment (e.g. 
maxmind)
- cgo dependency leading to: can't use a blank docker container, slower to 
compile and local installation dependencies
- something like site indexing for search does not share data so every 
concurrent instance builds at runtime after a deploy
- upgraded submodules may take a while to get to all the projects, leaving 
project feature inconsistent

I'm inclined to move out anything which has changing/largish data sets and 
centralize search (though during a rolling deploy this could be an 
interesting disaster).


On Saturday, March 11, 2017 at 10:05:49 PM UTC-8, Matt Ho wrote:
>
> If you're just starting out the project, by all means start with the 
> monolith where everything is together.  If you decide it's necessary, over 
> time you can split out the project into what you've discovered you need.  
>
> Trying to abstract the components you think you'll need too early is a 
> good recipe for heartache.  
>
> M
>
>
> On Saturday, March 11, 2017 at 9:00:43 PM UTC-8, James Pettyjohn wrote:
>>
>> I'm looking at the pros and cons of how to architect a web project.
>>
>> 1) One is a single go project for a site. No service dependencies for the 
>> backend at all. Certain aspects of this means a cgo dependency which is not 
>> ideal as it complicates containerization and slows build time. One plus to 
>> this is the simplicity in development - the entire project is self 
>> sufficient and tests are easily checking everything - thought compilation 
>> and startup suffer.
>>
>> 2) Another means would be to split out portions of the project. This adds 
>> dev complexity as it requires more services to enable parts of the server, 
>> or run at all. Coordination of service versions becomes an issue. But the 
>> services, now being centrally located, can be deployed and result in 
>> updates to all services using it. I think from the ops perspective this 
>> could be more efficient as you don't inherit the overhead in all projects.
>>
>>
>> Opinions on the above? Another approach entirely?
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Project architecture

[James Pettyjohn]

I'm looking at the pros and cons of how to architect a web project.

1) One is a single go project for a site. No service dependencies for the 
backend at all. Certain aspects of this means a cgo dependency which is not 
ideal as it complicates containerization and slows build time. One plus to 
this is the simplicity in development - the entire project is self 
sufficient and tests are easily checking everything - thought compilation 
and startup suffer.

2) Another means would be to split out portions of the project. This adds 
dev complexity as it requires more services to enable parts of the server, 
or run at all. Coordination of service versions becomes an issue. But the 
services, now being centrally located, can be deployed and result in 
updates to all services using it. I think from the ops perspective this 
could be more efficient as you don't inherit the overhead in all projects.


Opinions on the above? Another approach entirely?


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Notable Go 1.8 XML decoder bug fix

[James Pettyjohn]

I switched to go 1.8 today and found that some of my XML parsing was 
failing where it was passing in 1.6/1.7.

After a bit of searching I found that there was a bug that was *fixed* in 
1.8 and my faulty data was previously being ignored. Specifically empty 
attributes which mapped to non-nilable int types.

This was not mentioned in the release notes so I had trouble finding it, 
but did eventually find it was well documented in the issues preceding it 
the release:

https://github.com/golang/go/issues/16158 
https://go-review.googlesource.com/#/c/27455/


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Statically linking cgo

[James Pettyjohn]

Hi Justin,

Thanks for the holiday reply.

The ultimate goal in this was deployment into docker as a self contained 
binary, no dynamic library dependencies and no parent image. The most 
common solution to this I've seen is disable CGO altogether but with the 
libraries I'm using this is not an option at this time.

Best, 
James

On Saturday, December 24, 2016 at 5:36:48 PM UTC-8, Justin Israel wrote:
>
>
>
> On Sun, Dec 25, 2016 at 9:22 AM James Pettyjohn <japet...@gmail.com 
> > wrote:
>
>> I've been looking at a number of articles on statically linking go files 
>> by disabling CGO, sounds great except I've got a libsass dependency until 
>> the native libsass is ready.
>>
>> Can this be in a straightforward manner with go 1.6/1.7 on linux (centos)?
>>
>
> Are you asking about statically linking libsass?
>
> In addition to other flags to statically link your Go binary, you can tell 
> cgo to statically link libsass:
>
> $ export CGO_LDFLAGS="\
> -Wl,-Bstatic \
> `pkg-config --libs libsass` \
> -lotherDep -lotherDep2
> -Wl,-Bdynamic"
>
> ​
>
> - Justin
>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Statically linking cgo

[James Pettyjohn]

I've been looking at a number of articles on statically linking go files by 
disabling CGO, sounds great except I've got a libsass dependency until the 
native libsass is ready.

Can this be in a straightforward manner with go 1.6/1.7 on linux (centos)?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Increase speed of repeated builds

[James Pettyjohn]

No, it's some hundreds of strings across that whole package, none longer 
than 100 characters. No giant static XML strings or the like.

Thinking it might be telling to see dependency information I ran into 
(your) articles on golang dependency tools 
, so if it's of 
any note:

go list -f '{{ join .Imports "\n" }}' site_www2 | wc -l
  93
go list -f '{{ join .Deps "\n" }}' site_www2 | wc -l
 257

Don't know that this is very telling. 

Is there another important metric as far as go compilation? Or another 
compiler option to see numbers branches or the like which could point up 
something?


On Monday, August 15, 2016 at 1:40:19 PM UTC-7, Dave Cheney wrote:
>
> That looks like the case.
>
> Do you include a large amount of static data in the site_www2 package? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Increase speed of repeated builds

[James Pettyjohn]

I tried using 1.7rc6, no issues in compilation but finishes only about half 
a second under 1.6.2.

On Thursday, August 11, 2016 at 8:19:12 PM UTC-7, James Pettyjohn wrote:
>
> I noticed that in the release notes, I'll try out the RC and see what it 
> gives me.
>
> On Thursday, August 11, 2016 at 8:09:55 PM UTC-7, Michael Hudson-Doyle 
> wrote:
>>
>> On 12 August 2016 at 15:01, James Pettyjohn <japet...@gmail.com> wrote: 
>> > I have a project which has become fairly large, I simple check of *.go, 
>> no 
>> > _test files (including all modules), it's up to 1300 files. 
>> > 
>> > The main module though is 60 files, about 1MB of go source. It is 
>> taking to 
>> > about 20 seconds to compile for a single change in the main module 
>> using go 
>> > 1.6.2 darwin/amd64. 
>> > 
>> > Using -x on go install I'm seeing half the time on build and half the 
>> time 
>> > on linking. 
>> > 
>> > I think I've gotten all the direct CGO dependencies moved to submodules 
>> but 
>> > I'm not sure what would indicates if it is going through a cgo phase or 
>> not. 
>> > 
>> > Any input on other things to look for or tune on what is making the 
>> build 
>> > time so long? 
>>
>> 1.7 should be much faster (the linker is about 3 times faster on some 
>> things I tried). 
>>
>> Cheers, 
>> mwh 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Increase speed of repeated builds

[James Pettyjohn]

Maybe I wasn't clear, I am not sure what else to look at in this scenario. 
Whatever details you have in mind, i.e. which ones are relevant, is what 
I'm missing.

On Thursday, August 11, 2016 at 8:16:38 PM UTC-7, Dave Cheney wrote:
>
> Can you please post some details. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Increase speed of repeated builds

[James Pettyjohn]

I noticed that in the release notes, I'll try out the RC and see what it 
gives me.

On Thursday, August 11, 2016 at 8:09:55 PM UTC-7, Michael Hudson-Doyle 
wrote:
>
> On 12 August 2016 at 15:01, James Pettyjohn <japet...@gmail.com 
> > wrote: 
> > I have a project which has become fairly large, I simple check of *.go, 
> no 
> > _test files (including all modules), it's up to 1300 files. 
> > 
> > The main module though is 60 files, about 1MB of go source. It is taking 
> to 
> > about 20 seconds to compile for a single change in the main module using 
> go 
> > 1.6.2 darwin/amd64. 
> > 
> > Using -x on go install I'm seeing half the time on build and half the 
> time 
> > on linking. 
> > 
> > I think I've gotten all the direct CGO dependencies moved to submodules 
> but 
> > I'm not sure what would indicates if it is going through a cgo phase or 
> not. 
> > 
> > Any input on other things to look for or tune on what is making the 
> build 
> > time so long? 
>
> 1.7 should be much faster (the linker is about 3 times faster on some 
> things I tried). 
>
> Cheers, 
> mwh 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Increase speed of repeated builds

[James Pettyjohn]

I have a project which has become fairly large, I simple check of *.go, no 
_test files (including all modules), it's up to 1300 files. 

The main module though is 60 files, about 1MB of go source. It is taking to 
about 20 seconds to compile for a single change in the main module using go 
1.6.2 darwin/amd64.  

Using -x on go install I'm seeing half the time on build and half the time 
on linking.

I think I've gotten all the direct CGO dependencies moved to submodules but 
I'm not sure what would indicates if it is going through a cgo phase or not.

Any input on other things to look for or tune on what is making the build 
time so long?



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.