Re: [go-nuts] GC direct bitmap allocation and Intel Cache Allocation Technology

[Jesper Louis Andersen]

On Sun, Jul 31, 2016 at 11:47 AM, EduRam  wrote:

> It appears it will be possible to allocate partial CPU L3 cache to
> processes.


If I remember correctly, the original idea for the CAT technology was to
have "cache smashing" workloads on the same machines as other workloads.
The problem is that one workload fills up the cache and then when another
workload comes in, it has to pull everything into cache again. It was a
major reason to split work over multiple machines, and it really hurts in
virtualization environments.

My intuition somewhat follows what rlh@ writes: isolating a marking phase
over a heap so it can't evict every cache line sounds quite intriguing. In
fact, any "sweep" (not in the GC sense) over data could end up evicting
lots of useful cache lines which are accessed right after.



-- 
J.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: json murshal/unmarhsal struct that was got using custom interface type

[nathan.leclaire via golang-nuts]

Viacheslav,

Your best bet here in my opinion is to have an intermediate object that 
contains type information which you Unmarshal into first to get the 
concrete type.  Then, you have a switch to Unmarshal the remainder of the 
object into the real deal.  We did a similar mechanic to this on Docker 
Machine at one point where the various Drivers would contain a string 
indicating which type they "actually" are.

Nathan

On Monday, August 1, 2016 at 10:07:07 AM UTC-7, Viacheslav Biriukov wrote:
>
> Hi, 
>
> I want to marshal/unmarshal object for saving in key/value storage. 
> The question is: if I got this object using function, that accepts 
> interface type, lets say for example: base.Pricer. And there are more than 
> one type struct that satisfy this interface. I know while marshaling what 
> type this object have (using reflection), and theoretically can save this 
> type in storage. But when I try to load this object – I want to get exactly 
> that type what was before serialization. But I don't know how to do this 
> and is this is a good pattern?
>
> Thank you.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Using golang/cmd/pprof/internal in a Go program, and/or UNIX socket support in `go tool pprof`?

[Brad Fitzpatrick]

It used to be. It was rewritten in Go.


On Mon, Aug 1, 2016 at 10:44 AM, Nathan LeClaire  wrote:

> Oh, interesting call out.  Thanks Brad.  For some reason I assumed pprof
> was a more general-purpose tool written in C++.  Forget where I may have
> read that.
>
> On Mon, Aug 1, 2016 at 10:14 AM, Brad Fitzpatrick 
> wrote:
>
>> Note that Go's "go tool pprof" is basically just
>> https://github.com/google/pprof
>>
>> You can vendor that into the Docker daemon and have the server profile
>> itself and send the all-in-one output over the unix socket to the docker
>> command line tool, which users can then run and file a bug report with the
>> output.
>>
>>
>>
>> On Mon, Aug 1, 2016 at 10:10 AM, Nathan LeClaire <
>> nathan.lecla...@docker.com> wrote:
>>
>>> Hey Brad!
>>>
>>> Yes, end users.  Generally, we can't make assumptions about what they
>>> have installed locally (e.g. Go toolchain) and I would like to provide a
>>> way for folks to "click a button" and get a fully symbolized profile to
>>> send to us.  Doing this inside of an existing Go program would be ideal.
>>>
>>> On Sat, Jul 30, 2016 at 1:34 PM, Brad Fitzpatrick 
>>> wrote:
>>>
 Who is your target audience for this?

 You seem to know how to do it (socat + go tool pprof), which suggests
 you want end users to do this or something?


 On Fri, Jul 29, 2016 at 3:43 PM, nathan.leclaire via golang-nuts <
 golang-nuts@googlegroups.com> wrote:

> Hi all,
>
> I am interested in doing performance profiling on the Docker daemon
> using the existing pprof tools and/or code inside of the internal packages
> of commands, and I was hoping to get some guidance on the challenges I've
> encountered attempting this.
>
> The Docker daemon (a Go program) exposes the pprof endpoints at
> /debug/pprof.  However, by default the Docker daemon only listens on
> a Unix domain socket to expose its HTTP API, and exposing it over a
> non-encrypted TCP port is generally inadvisable due to privilege 
> escalation
> concerns.
>
> The current most common method for accessing this pprof information
> seems to be to use socat to temporarily forward requests from the socket 
> to
> a locally listening TCP port, and use go tool pprof to collect
> profile information and analyze it.  This works OK for local development,
> but I have a few questions about how we might be able to expand support 
> for
> collecting these pprof dumps and analyzing them more easily:
>
> 1. Would a proposal be considered to add support for collecting this
> information directly through go tool pprof , e.g. go tool pprof
> unix:///var/run/docker.sock, or is it not an area of interest for the
> Go tools?  Some possible dilemmas include the unix:// protocol
> convention, which seems to be fairly Docker-unique to me and a little odd
> to conflate (transport layer vs. protocol) with http://.  I've looked
> extensively at the code and it doesn't seem to cover this today.
> 2. Is it possible to install and use go tool pprof in a minimal
> manner, i.e. without the rest of the Golang toolchain?  If so, how?
> 3. How inadvisable would it be to use the internal code for generating 
> *Profile
> and symbolizing the profiles in a 3rd party program?  Obviously due to
> the internal it's not meant to be exported but it would be very nice
> to be able to directly embed this type of code in a library-like fashion 
> to
> be able to quickly generate dumps from running daemons that could later be
> loaded with rich semantic information into go tool pprof on another
> computer (without also needing the source binary).  Naturally it's
> *possible* to just cp and vendor the code from the stdlib and work
> around this restriction, but is it advisable?
> 4. Any other ideas for getting a easily importable stand-alone *.pb.gz
> pprof output from inside of an exiting Go program (separate from the one
> that is being profiled)?  I had an idea to make a minimal Go program (or
> embed in an existing one) which might be quite good at this, but getting
> richly annotated information (including symbols, etc.) via HTTP alone
> without any of the surrounding internal code to process it has proven a 
> lot
> trickier than I naively assumed at first.
>
> Thanks all, and thanks of course for go tool pprof in the first
> place, it's a really excellent tool.
>
> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>


>>>
>>
>

-- 
You received this 

Re: [go-nuts] Using golang/cmd/pprof/internal in a Go program, and/or UNIX socket support in `go tool pprof`?

[Brad Fitzpatrick]

Note that Go's "go tool pprof" is basically just
https://github.com/google/pprof

You can vendor that into the Docker daemon and have the server profile
itself and send the all-in-one output over the unix socket to the docker
command line tool, which users can then run and file a bug report with the
output.



On Mon, Aug 1, 2016 at 10:10 AM, Nathan LeClaire  wrote:

> Hey Brad!
>
> Yes, end users.  Generally, we can't make assumptions about what they have
> installed locally (e.g. Go toolchain) and I would like to provide a way for
> folks to "click a button" and get a fully symbolized profile to send to
> us.  Doing this inside of an existing Go program would be ideal.
>
> On Sat, Jul 30, 2016 at 1:34 PM, Brad Fitzpatrick 
> wrote:
>
>> Who is your target audience for this?
>>
>> You seem to know how to do it (socat + go tool pprof), which suggests you
>> want end users to do this or something?
>>
>>
>> On Fri, Jul 29, 2016 at 3:43 PM, nathan.leclaire via golang-nuts <
>> golang-nuts@googlegroups.com> wrote:
>>
>>> Hi all,
>>>
>>> I am interested in doing performance profiling on the Docker daemon
>>> using the existing pprof tools and/or code inside of the internal packages
>>> of commands, and I was hoping to get some guidance on the challenges I've
>>> encountered attempting this.
>>>
>>> The Docker daemon (a Go program) exposes the pprof endpoints at
>>> /debug/pprof.  However, by default the Docker daemon only listens on a
>>> Unix domain socket to expose its HTTP API, and exposing it over a
>>> non-encrypted TCP port is generally inadvisable due to privilege escalation
>>> concerns.
>>>
>>> The current most common method for accessing this pprof information
>>> seems to be to use socat to temporarily forward requests from the socket to
>>> a locally listening TCP port, and use go tool pprof to collect profile
>>> information and analyze it.  This works OK for local development, but I
>>> have a few questions about how we might be able to expand support for
>>> collecting these pprof dumps and analyzing them more easily:
>>>
>>> 1. Would a proposal be considered to add support for collecting this
>>> information directly through go tool pprof , e.g. go tool pprof
>>> unix:///var/run/docker.sock, or is it not an area of interest for the
>>> Go tools?  Some possible dilemmas include the unix:// protocol
>>> convention, which seems to be fairly Docker-unique to me and a little odd
>>> to conflate (transport layer vs. protocol) with http://.  I've looked
>>> extensively at the code and it doesn't seem to cover this today.
>>> 2. Is it possible to install and use go tool pprof in a minimal manner,
>>> i.e. without the rest of the Golang toolchain?  If so, how?
>>> 3. How inadvisable would it be to use the internal code for generating 
>>> *Profile
>>> and symbolizing the profiles in a 3rd party program?  Obviously due to
>>> the internal it's not meant to be exported but it would be very nice to
>>> be able to directly embed this type of code in a library-like fashion to be
>>> able to quickly generate dumps from running daemons that could later be
>>> loaded with rich semantic information into go tool pprof on another
>>> computer (without also needing the source binary).  Naturally it's
>>> *possible* to just cp and vendor the code from the stdlib and work
>>> around this restriction, but is it advisable?
>>> 4. Any other ideas for getting a easily importable stand-alone *.pb.gz
>>> pprof output from inside of an exiting Go program (separate from the one
>>> that is being profiled)?  I had an idea to make a minimal Go program (or
>>> embed in an existing one) which might be quite good at this, but getting
>>> richly annotated information (including symbols, etc.) via HTTP alone
>>> without any of the surrounding internal code to process it has proven a lot
>>> trickier than I naively assumed at first.
>>>
>>> Thanks all, and thanks of course for go tool pprof in the first place,
>>> it's a really excellent tool.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] const values of any types can't be compared to interface{} values?

[T L]

ex:

package main
>
> func main() {
> // error: illegal constant expression: *int == interface {}
> _ = (*int)(nil) == interface{}(nil)
> }
>


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: GC direct bitmap allocation and Intel Cache Allocation Technology

[rlh]

Thanks for pointing this out. While it isn't clear how this is applicable 
to the sweep free alloc 
work it does seem relevant to the mark worker's heap tracing which can 
charitable be 
described as a cache smashing machine. The mark worker, loads an object 
from a random 
location in memory,  scans it, drops pointers to other objects needing to 
be scanned into a 
buffer, and then does not visit the object's cache lines again. The loads 
evicts cache lines 
being used by goroutines and while a victim cache may help ameliorate the 
problem any 
cost we can move from a goroutine to the GC worker is a win.

Segregating the GC's cache lines from that of the go-routines should result 
in the 
goroutine's cache lines not being evicted and the GC evicting a cache line 
recently used by
the GC. This seems like a good replacement policy. It certainly seems like 
a promising avenue. 
It's a build and measure. (and build and measure)


 

On Sunday, July 31, 2016 at 5:47:44 AM UTC-4, EduRam wrote:
>
> Hi!
>
> I am catching up some summer readings ... and recently came across this 
> article 
> about Intel Cache Allocation Technology (CAT), on Haswell processors
> (http://lwn.net/Articles/694800/)
>
>
> It appears it will be possible to allocate partial CPU L3 cache to 
> processes.
>
>
> I remember reading about a new experiment on Go runtime GC allocation, 
> that would use more "cache friendly" bitmap.
> (
> https://github.com/golang/proposal/blob/master/design/12800-sweep-free-alloc.md
> )
>
>
> Just out of curiosity ... could this CAT mechanism, help further the GC 
> mechanism ?
> I must confess this is outside my knowledge domain. Just reading for fun 
> of it.
>
> Thanks and great holidays,
>
> Edu
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: How Go GC perform with 128GB+ heap

[Jesper Louis Andersen]

On Mon, Aug 1, 2016 at 6:38 AM, 'Sugu Sougoumarane' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> Instead, if you ran something like 10x instances of the same program,
> you'll end up using the same resources, but will get much better
> performance, and your code will be much simpler.


This is the Erlang solution to the problem. For each proxy-request, run a
server, in isolation. You don't run one process handling a million
requests. You run a million processes each handling one request. It is
literally what happens in the VM internals as well. Each process has its
own heap and can be GC'ed in isolation. Of course this requires you copy
data into the process for communication, and this is the secret of the
Erlang VM. In fact, everyone else is doing it wrong, often citing odd
claims of "efficiency" as a reason. You end up having to build rather
complex GC schemes, whereas Erlang just uses a tried and true 2-space
copying collection by Cheney.


-- 
J.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: How Go GC perform with 128GB+ heap

[rlh]

I think the high bit here is that the Go community is very aggressive about 
GC latency. Go has large
users with large heaps, lots of goroutines, and SLOs similar to those being 
discussed here. When 
they run into GC related latency problems the Go team works with them to 
root cause and address
the problem. It has been a very successful collaboration. 

Stepping back, work on large heaps is being motivated by the fact that RAM 
hardware, due to its
thermal characteristics, is still doubling byte/$ every 2 years or so. As 
heaps grow GC latency needs
to be independent of heap size if Go is going to continue to scale over the 
next decade. The Go 
team is well aware of this, is motivated by it, and continues to design a 
GC to address this trend. 

On Sunday, July 31, 2016 at 9:26:13 AM UTC-4, almeida@gmail.com wrote:
>
> I'm starting a proof of concept project to the company i work. The project 
> is a http proxy with a smart layer of cache (Varnish, Nginx and others 
> don't work because we have business rules on cache invalidation) for a very 
> big microservice architecture (300+ services).
>
> We have 2x128GB machines available today for this project. 
> I don't have any doubt that Go has amazing performance, used in other 
> projects, and they are rock solid, very fast and consuming very little 
> memory.
> But i'm afraid to use Go at this project because of the GC. I'm planning 
> to use all the available memory on cache. Isn't all this memory on heap be 
> a problem?
>
> It's a new area to me, store tons of GB in a GC language.
> What is my options? Use a []byte and or mmap to stay out of GC?
> Lots and lots of code to reimplement this datastructures on top of slices 
> just to avoid the GC, not counting all the encoding/decoding to get/set the 
> values.
>
> Stick with the raw slices?
> Didn't used Cgo before, but it is a viable option? 
> Or should i go 100% offheap with something like Rust or C?
>
> I hope to add as little overhead as possible.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: html/template function not defined

[Matt Silverlock]

Where is the code for parsing your templates and apply the FuncMap?

On Saturday, July 30, 2016 at 7:05:39 AM UTC-7, 
alex.b...@leadinglocally.com wrote:
>
>
> Since, I have made 2 more float-to-int functions and they work fine.
>
> To solve this I'm having to pre-marshal my json before I execute the 
> template.
>
> Please if anyone knows why it doesn't like JSON encoding functions in the 
> templates, let me know
>
>
> On Saturday, July 30, 2016 at 4:32:12 AM UTC+2, 
> alex.b...@leadinglocally.com wrote:
>>
>> func encodeJSON(data interface{}) template.JS {
>> a, err := json.Marshal(data)
>> if err != nil { panic(err) }
>>
>> return template.JS(a)
>> }
>>
>> // code for rendering ELEMENT
>>
>> func (ele *ELEMENT) Render() ([]byte, error) {
>>
>> // init map of functions available to use within delimiters
>>
>> if funcMap == nil {
>> funcMap = template.FuncMap{
>> "divideFF": divideFF,
>> "percentageFF": percentageFF,
>> "percentageIF": percentageIF,
>> "encodeJSON": encodeJSON,
>> }
>> }
>>
>> when executing only the encodeJSON function I get this: template: x:19: 
>> function "encodeJSON" not defined
>>
>> all others work fine, and I checked the correct code is in my gopath.
>>
>> please advise... I had the encodeJSON func returning a string before but 
>> it didn't work either
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] How Go GC perform with 128GB+ heap

[Chris Randles]

Consul and etcd use Boltdb (https://github.com/boltdb/bolt) which uses an 
mmap-ed file.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] gomobile installation problem

[vejdani . shojae]

Hi all dears
I'm beginner in golang and i interested to install gomobile and test it.
unfortunately when i going to install gomobile using:
 "go get golang.org/x/mobile/cmd/gomobile"
 it broken and give  this error massage:

package golang.org/x/mobile/cmd/gomobile: unrecognized import path "
golang.org/x/mobile/cmd/gomobile" (parsing golang.org/x/mobile/cmd/gomobile: 
XML syntax error on line 9: expected /> in element)

please help me... how can i install go mobile ...

thanks

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] How Go GC perform with 128GB+ heap

[Sokolov Yura]

Go 100 off-heap.

You may use other in-memory database for data, running on a same machine. I 
recomend Tarantool: http://tarantool.org - it is capable to handle hundreds of 
thousands (up to million) requests per second on just one CPU core. If you need 
more, then you may consider sharding. If you need no persistency, you may 
disable logging.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.