Re: [go-nuts] Re: REST API - Request Logging on Server

[desaiabhijit]

Hi Jesper

Thanks for the reply

Do you need to log every request? 
yes.. supporting minimum 200 req/sec per machine

Do you need to know the contents of the request? 
yes.. it's a search request having 6 parameters ( XML/JSON REST )

Need to log all the requests for BI purpose so can't limit logging

Rgds,

Abhi


On Monday, July 17, 2017 at 11:17:43 PM UTC+5:30, Jesper Louis Andersen 
wrote:
>
> Do you need to log every request? Do you need to know the contents of the 
> request? Metrics are far cheaper to maintain than a log line. Also, 
> consider sampling, or only logging slow requests (though put a limit on 
> them over a period: when your system goes wrong you don't want all the slow 
> logging requests be the bottleneck in the system).
>
> On Mon, Jul 17, 2017 at 7:03 PM  wrote:
>
>> Hi Nate
>>
>> Thanks very much for the info
>>
>> Rgds,
>>
>> Abhi
>>
>>
>>
>> On Monday, July 17, 2017 at 10:19:58 PM UTC+5:30, Nate Finch wrote:
>>>
>>> Basically any log package will be pretty quick.   None of them will be 
>>> zero overhead, that's impossible, but most will not affect the time of your 
>>> requests terribly significantly.   
>>>
>>> I personally like logrus (https://github.com/sirupsen/logrus) for 
>>> structured logging, it has a lot of adapters available to pipe the log 
>>> wherever you need it to go.  There's also a middlewear for gin called 
>>> ginrus that'll log to logrus.
>>>
>>> If you want the fastest possible logging, glog (
>>> https://github.com/golang/glog) is very optimized but is just 
>>> plaintext, not structured, and doesn't have all the adapters of logrus.
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Programming paradigm.

[Patrick Logan]

If you want to do FP and only FP then you will be heartsick for an ML-like 
language and should use one of those. However there is a lot of room between 
"no FP" and "only FP"... call this "semi-functional" and Go accommodates this 
well.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Programming paradigm.

[Henry]

This has me piqued. How do you do recursion without tail call optimization? As 
far as I know, Go compiler doesn't do tail call optimization. The heart of FP 
lies in the composability of function. Go supports higher order function, so 
you can compose functions albeit in a limited way. Without generics and 
function currying, however, it far from what the real FP can do. In addition, 
FP compiler is a sophisticated compiler with aggressive optimization whereas Go 
compiler is a WYSIWYG compiler with micro optimization. Even if you can emulate 
FP in Go, the resulting performance will be worse than if you would use a real 
FP language. So I don't think it is practical to do an actual FP in Go. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] encoding/csv: Is this a bug?

[Matt Harden]

I suspect that this has to do with the line-ending characters on a Mac. I
think Excel is writing the file with each line ending with a CR character.
The encoding/csv package expects RFC 4180 format (each line terminated with
CRLF), which is what Excel writes when you select "Windows Comma Separated".

I don't know a super-easy way to make encoding/csv accept the first format.

On Mon, Jul 17, 2017 at 4:42 PM Dat Huynh  wrote:

> Hi all,
>
> I have a problem with parsing a .csv file using the library "encoding/csv".
>
> I wonder if that is the problem of Microsoft Excel or the Go library.
>
> I am using Microsoft Excel version 14.2.2 on MacOS and go1.8.3 darwin/amd64
>
> What did I do?
>
> Firstly I input the below values into an Excel sheet, and save as a .csv
> file.
> value 11 value 12
> value 21 value 22
> value 31 value 32
>
> If I choose "Comma Separated Values (.csv)" in the option "Format", type
> the file name "data.csv", and run my Go app, it returns:
>
> $ go run demo.go
> value 31 value 32]12
>
> If I choose "Window Comma Separated (.csv)" in the option "Format", type
> the file name "data.csv", and run my Go app, it works well.
>
> $ go run demo.go
> 0 [value 11 value 12]
> 1 [value 21 value 22]
> 2 [value 31 value 32]
>
> Could you please confirm if this is a bug of the library or MS Excel?
>
> Below is my code.
>
> package main
>
> import (
> "encoding/csv"
> "fmt"
> "os"
> )
>
> func main() {
> file, _ := os.Open("data.csv")
> defer file.Close()
> csvReader := csv.NewReader(file)
> records, _ := csvReader.ReadAll()
> for index, record := range records {
> fmt.Println(index, record)
> }
> }
>
> Thank you very much.
>
> Regards,
> Dat Huynh.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] An idea about motivating the community

[Nate Finch]

(sorry for the duplicate post, the other one disappeared)

I wrote a tweet after Gophercon about making a resolution to write more 
blog posts.  I used the hashtag #GopherResolution with the hope that other 
people might pick up the idea and run with it.  So far, however, only one 
other person has used the tag.  

There are studies on the effects of making public promises (resolutions) to 
act that indicate it is generally effective in getting you to stick to 
those resolutions (at least more so than ones you don't announce).

However, it occurred to me that most resolutions fail because they're too 
vague.  There's an acronym called SMART that says your goals should be 
Specific, Measurable, Achievable, Realistic, and Time-based.  The 
"measurable" one is the interesting one, and where I think we can leverage 
software to help us.  

My idea is to make a website where gophers can make specific resolutions to 
contribute to the community, and the website can then track those 
resolutions over time.  For example, a resolution to write blog posts could 
be tracked by consuming an RSS Feed.  A resolution to contribute to a 
particular project could track PRs, issues, etc.  The website could track 
streaks over time if we give a time box for each contribution (write a blog 
post every 2 weeks, or contribute to one open source project every month).

With dashboards showing top contributors, etc, it could effectively 
game-ify community involvement.

Of course, all of this would take a significant amount of work to code, but 
it seems very doable, if enough people would actually participate.

Thoughts?  Is this something you'd be interested in?  Do you think it would 
be valuable to build?

-Nate

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: [Go2] toward-go2

[Ian Lance Taylor]

On Mon, Jul 17, 2017 at 5:13 PM, Gert  wrote:
>
> Let me ask it differently what is going to happen with all the Go2 labeled
> issues on github?

They will be considered over time as we try to figure out what might
make sense for Go 2.


> For me experience report is nothing different then a
> github issue

Well, no.  As Russ explained in the Go 2 blog post, the Go programming
language has always been developed in a pragmatic way based on actual
experience using Go.  In asking for experience reports, we are asking
people to explain specific ways that the Go programming language is
not working well for their actual work.  This can of course be done in
the form of a github issue, but most github issues are not that.


> and also nobody of the go team will be able to handle all the
> incoming false positives of people like me that think have a great idea but
> actually isn't or already has been addressed. Sorry I just prefer the other
> way around, the go team makes a report looking at golang code on github and
> question on stackoverflow and make proposal to clean up standard library and
> improve the language.

It is likely that something along those lines will happen.  But the
result will be better if we first get information about problems that
people run into while writing real Go code.  Of course we can collect
our own set of problems, but the Go community is much much much larger
than the Go team.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: [Go2] toward-go2

[Gert]

Let me ask it differently what is going to happen with all the Go2 labeled 
issues on github? For me experience report is nothing different then a 
github issue and also nobody of the go team will be able to handle all the 
incoming false positives of people like me that think have a great idea but 
actually isn't or already has been addressed. Sorry I just prefer the other 
way around, the go team makes a report looking at golang code on github and 
question on stackoverflow and make proposal to clean up standard library 
and improve the language. The same way to go team did with Go 1.9 and type 
aliases. Reporting the issue listen to feedback and implement it. Go2 
should be a breath of fresh air to finally undo a few mistakes and clean 
things up, it should definitely not be a publicity stunt and just leave all 
the mistakes in to be backward compatible.

On Monday, July 17, 2017 at 7:06:15 AM UTC+2, Nathan Kerr wrote:
>
> The active work for Go2 right now is gathering experience reports 
> . Most of the changes resulting 
> from these reports will land in Go1.x. It is expected that Go2 will only 
> have a few major changes that will not be compatible with Go1. It is also 
> possible there will be none (and Go1.20 will be called Go2)
>
> The Go2 label in the issue tracker has meant, more or less, that the 
> issue/proposal was something that could't be considered for Go1, usually 
> because it would be the Go1 compatibility promise, but should be kept track 
> of.
>
> Apple didn't mention Swift because Apple doesn't mention things it is 
> working on. Working in private and then releasing when things are ready is 
> how Apple does things.
>
> On Monday, July 17, 2017 at 5:42:34 AM UTC+2, Gert wrote:
>>
>> There are so many 
>> https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+go2+label%3AGo2 
>> proposals already I can barely figure out if feature x y or z is already 
>> asked.
>>
>> Can we have a Go2 branch / milestone please? PS now that you announced 
>> toward-go2 you are going to actively work on Go2 right? Its not some kind 
>> of lets try to trick developers to keep using Go who are looking for 
>> alternatives because they made a Go2 proposal many years ago and gave up 
>> hope right? Don't get me wrong I love Go but I hate false promises or 
>> making plans ridiculous far in the future. Not saying Go2 can't take a long 
>> time to make, just saying i would like to see small bits of code emerging 
>> on a weekly bases we can test and play around with and give feedback on 
>> real things not just imaginary ideas :) If you can't make a Go2 branch yet 
>> and more important things need to be done first on Go1 I understand 
>> completely, but once you pull the Go2 card companies will pull back and 
>> start waiting for Go2. Its the number one reason Apple didn't mention a 
>> single word about Swift until they could deliver a fully functioning 
>> compiler just because of that reasons to not jeopardise developers stop 
>> learning objective c and waiting for Swift
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] encoding/csv: Is this a bug?

[Dat Huynh]

Hi all,

I have a problem with parsing a .csv file using the library "encoding/csv".

I wonder if that is the problem of Microsoft Excel or the Go library.

I am using Microsoft Excel version 14.2.2 on MacOS and go1.8.3 darwin/amd64

What did I do?

Firstly I input the below values into an Excel sheet, and save as a .csv
file.
value 11 value 12
value 21 value 22
value 31 value 32

If I choose "Comma Separated Values (.csv)" in the option "Format", type
the file name "data.csv", and run my Go app, it returns:

$ go run demo.go
value 31 value 32]12

If I choose "Window Comma Separated (.csv)" in the option "Format", type
the file name "data.csv", and run my Go app, it works well.

$ go run demo.go
0 [value 11 value 12]
1 [value 21 value 22]
2 [value 31 value 32]

Could you please confirm if this is a bug of the library or MS Excel?

Below is my code.

package main

import (
"encoding/csv"
"fmt"
"os"
)

func main() {
file, _ := os.Open("data.csv")
defer file.Close()
csvReader := csv.NewReader(file)
records, _ := csvReader.ReadAll()
for index, record := range records {
fmt.Println(index, record)
}
}

Thank you very much.

Regards,
Dat Huynh.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Can't explain Golang benchmarks

[Jesper Louis Andersen]

On Mon, Jul 17, 2017 at 10:14 PM Zohaib Sibte Hassan <
zohaib.has...@gmail.com> wrote:

> So the scenario I am trying to implement is basically a chat scenario.
>

My first approach would be to measure.

* The full system will have a bottleneck as per Amdahl's law. You need to
take a holistic view since the bottleneck may be outside Go (ie, the
kernel, hardware, ...). Knowing the bottleneck will usually suggest an
angle of attack on the problem at hand.

* High memory usage suggests either an overload situation (because you
don't have enough resources) or excessive copying of data is going on. Try
doing some napkin math. A million active users with 1k of data each takes
one gigabyte of memory and so on.

* High CPU can be indicative of real work, that the CPU is bandwidth
constrained toward memory, or that lock contention is going on. Kernels
usually will not discriminate between these states, so some investigation
is necessary.

* Keeping thousands of connections on small hardware is expensive. Each TCP
connection needs some kernel space in addition to userland space. It
quickly adds up.

* TCP sending is going to cost a lot of time. In laboratory tests, the
network is fast. Inside a datacenter, such as one operated by Google,
network transmission is fast. The internet in general is slow,
latency-inducing, and brittle. This forces your system to keep data
lingering for longer, and this puts more pressure on memory usage.

* Observation: in a noisy chatroom, you want to skip messages if they flow
out of the view on the client. You don't need to process every message in
this case. Just the messages that are visible. This suggests a polling
construction like the disruptor: only care about the K newest messages in
the fast path. The slow path does historical lookups. Keep an "epoch" count
of where we are in the message flow. When a socket is ready for data, use
the epoch count to figure out what happened in the meantime.

* If messages are immutable, they don't change and can be concurrently read
with little overhead. Edits to messages can be handled by a
patching-construction which overrides an earlier message.

* The *publisher* should take the effort of constructing as much of the
payload as possible and place it into a buffer everyone writes directly
into the network socket. If every subscriber has to do work, things get
expensive.

* Channels are likely to be fast enough, and resources are likely to run
out quickly: especially memory on a 512 megabyte computer.

* Channels should be used to send epochs around. The actual payload ought
to be somewhere else where it is ready and barriered appropriately for
read-only consumption. Alternative: pass a reference to the data around.
This is simple to do and is likely to be fast.

* Your goal is to get data to the socket so the kernel can do work. If you
do this correctly, it is likely the socket is going to be the bottleneck of
the system. Also, consider the possibility that you will block a goroutine
on data transfer to the outside world while its channel buffer fills up.
The publisher should not block in this situation.

In general, system engineering tend to trump local tuning. Effort is a
constrained resource, so it is usually best spent in the areas where the
cost/benefit analysis falls out nicely.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Error: opcode not supported on this processor: mips32

[Ian Lance Taylor]

On Mon, Jul 17, 2017 at 12:53 PM, Sander van Harmelen  wrote:
>
> I’m trying to cross compile
> https://github.com/google/gousb/tree/master/lsusb from macOS to mipsle using
> the following cmd:
>
>> GOOS=linux GOARCH=mipsle CGO_ENABLED=1 CC=mipsel-openwrt-linux-gcc
>> CFLAGS=-I../include/ go build
>
> But I receive this error:
>
> # runtime/cgo
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S: Assembler messages:
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:30: Error: opcode not supported on
> this processor: mips32 (mips32) `sdc1 $f20,40($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:31: Error: opcode not supported on
> this processor: mips32 (mips32) `sdc1 $f22,48($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:32: Error: opcode not supported on
> this processor: mips32 (mips32) `sdc1 $f24,56($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:33: Error: opcode not supported on
> this processor: mips32 (mips32) `sdc1 $f26,64($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:34: Error: opcode not supported on
> this processor: mips32 (mips32) `sdc1 $f28,72($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:35: Error: opcode not supported on
> this processor: mips32 (mips32) `sdc1 $f30,80($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:52: Error: opcode not supported on
> this processor: mips32 (mips32) `ldc1 $f20,40($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:53: Error: opcode not supported on
> this processor: mips32 (mips32) `ldc1 $f22,48($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:54: Error: opcode not supported on
> this processor: mips32 (mips32) `ldc1 $f24,56($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:55: Error: opcode not supported on
> this processor: mips32 (mips32) `ldc1 $f26,64($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:56: Error: opcode not supported on
> this processor: mips32 (mips32) `ldc1 $f28,72($29)'
> /usr/local/go/src/runtime/cgo/gcc_mipsx.S:57: Error: opcode not supported on
> this processor: mips32 (mips32) `ldc1 $f30,80($29)'
>
> I’ve build mipsel-openwrt-linux-gcc using the docs written down here:
> https://wiki.openwrt.org/doc/howto/build
>
> Any pointers on how to make this work? Or am I out of luck and did I hit
> road block (i.e. it’s not possible/supported)?

I don't understand that error, as to the best of my knowledge the
mips32 ISA supports the ldc1 and sdc1 instructions.

Is it possible that you need to update your GNU binutils?

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Deleted post?

[Wojciech S. Czarnecki]

On Mon, 17 Jul 2017 13:52:17 -0700 (PDT)
Nate Finch  wrote:

> I guess my previous post was deleted? Can a mod contact me about it please?
> 

Recently I got into my inbox yours:
> From: Nate Finch 
> To: golang-nuts 
> Subject: [go-nuts] An idea about motivating the community
> Date: Mon, 17 Jul 2017 04:58:50 -0700 (PDT)

Though I remember cases of some posts not appearing on web gg abysmal
interface and I recall advice in line of "Try to flush your gg cookies, clear
cache and possibly start browser afresh - missing post should reappear".

Hope this helps,

-- 
Wojciech S. Czarnecki
 << ^oo^ >> OHIR-RIPE
(I'm not a mod  ;)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Deleted post?

[Nate Finch]

I guess my previous post was deleted? Can a mod contact me about it please?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] slow Mod function in math/big, a better way?

[Rémy Oudompheng]

2017-07-17 15:56 GMT+02:00 Jeff Templon :
> Coming back to this, now that my Go knowledge is increasing:
>
> On Saturday, July 8, 2017 at 5:58:59 PM UTC+2, Andy Balholm wrote:
>>
>> I noticed your “naive explanation” after I sent my message. But I think it
>> is the real explanation.
>
>
> it turns out that 77% of the time of the program is spent in
> runtime.mallocgc :-)  I think the reason why is stuff like this:
>
> func (z nat) shr(x nat, s uint) nat {
>   m := len(x)
>   n := m - int(s/_W)
>   if n <= 0 {
>   return z[:0]
>   }
>   // n > 0
>
>   z = z.make(n)
>   shrVU(z, x[m-n:], s%_W)
>
>   return z.norm()
>
>   }
>
> If I understand it right, there are two "make slice" operations in this
> code, and this bit is being run billions of times (heart of a tight loop).
> Given the way Go works, the GC is unavoidable.  I've managed to make the
> code slightly faster by both swapping right-shift in instead of a Mod
> (Barrett reduction) and also avoiding my own object creation inside the
> loop, and there is 10 to 20% to be won by playing with GOGC, but the only
> way to do better is probably to rewrite the math/big library.  I tried using
> GMP but given all the temporaries created, it was even worse than using
> math/big!
>

How are you writing your code ? Wisely using variables should only
cause very minimal memory allocation.

Rémy.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] slow Mod function in math/big, a better way?

[Jan Mercl]

On Mon, Jul 17, 2017 at 7:18 PM Jeff Templon 
wrote:

> Sure:

I hoped for a number of which you don't know the factors and wanted to
cheat by linking to factordb.com ;-)

Earlier you wrote also

> it turns out that 77% of the time of the program is spent in
runtime.mallocgc :-)  I think the reason why is stuff like this:

I wonder if you happen to use Linux/amd64 and if you want to try (published
right now): https://github.com/cznic/minigmp. If so, please share the
performance changes, if any, thank you.

-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Can't explain Golang benchmarks

[Michael Jones]

I get about 3 million channel send/receive pairs per second. If you
actually do work inbeteeen (database access, computation, etc.) the limit
is difficult to reach.

On Mon, Jul 17, 2017 at 10:22 AM Jesper Louis Andersen <
jesper.louis.ander...@gmail.com> wrote:

> Your benchmarks are not really *doing* something once they get data. This
> means that eventual conflicts on (internal or external) locks are going to
> be hit all the time. In turn, you are likely to measure the performance of
> your hardware in the event of excessive lock contention (this hypothesis
> can be verified: block profiling).
>
> Were you to process something before going back to the barrier or channel,
> then you are less likely to hit the problem. Though you should be aware
> sometimes your cores will coordinate (because the work they do will tend
> them to come back to the lock at the same point in time - a similar problem
> to TCP incast).
>
> As you go from a single core to multiple cores, an important event happens
> in the system. In a single-core system, you don't need the hardware to
> coordinate. This is much faster than the case where the hardware might need
> to tell other physical cores about what is going on. Especially under
> contention. In short, there is a gap between the single-core system and a
> multicore system due to communication overhead. In high-performance
> computing a common mistake is to measure the *same* implementation in the
> serial and parallel case, even though the serial case can be made to avoid
> taking locks and so on. A fair benchmark would compare the fastest
> single-core implementation with the fastest multi-core ditto.
>
> The upshot of having access to multiple cores is that you can get them all
> to do work and this can lead to a substantial speedup. But there has to be
> work to do. Your benchmark has almost no work to do, so most of the cores
> are just competing against each other.
>
> For a PubSub implementation you need to think about the effort involved in
> the implementation. Up to a point, I think a channel-based solution, or a
> barrier based solution is adequate. For systems where you need low latency,
> go for a pattern like the Disruptor pattern[0], which should be doable in
> Go. This pattern works by a ring-buffer of events and two types of "clock
> hands". One hand is the point of insertion of a new event, the other
> hand(s) track where the readers currently are in their processing. By
> tracking the hands through atomics, you can get messaging overhead down by
> a lot. The trick is that each thread writes to its own hand and other
> threads can read an eventually consistent snapshot. As long as the buffer
> doesn't fill up completely, the eventually consistent view of an ever
> increasing counter is enough. Getting an older value is not a big deal, as
> long as the value isn't undefined.
>
> The problem with channels are that they are, inherently, one-shot. If a
> message is consumed, it is gone from the channel. In a PubSub solution, you
> really want multiple readers process the same (immutable) data and flag it
> when it is all consumed. Message brokers and SMTP servers often optimize
> delivery of this kind by keeping the payload separate from the header
> (envelope): better not pay for the contents more than once. But you still
> have to send a message to each subscriber, and this gets rather expensive.
> The Disruptor pattern solves this.
>
> On the other hand, the Disruptor only works if you happen to have memory
> shared. In large distributed systems, the pattern tend to break down, and
> other methods must be used. For instance a cluster of disruptors. But then
> keeping consistency in check becomes a challenge of the fun kind :)
>
> [0] http://lmax-exchange.github.io/disruptor/files/Disruptor-1.0.pdf -
> Thompson, Farley, Barker, Gee, Steward, 2011
>
> On Mon, Jul 17, 2017 at 7:46 AM Zohaib Sibte Hassan <
> zohaib.has...@gmail.com> wrote:
>
>> Awesome :) got similar results. So channels are slower for pubsub. I am
>> surprised so far, with the efficiency going down as the cores go up (heck
>> reminds me of that Node.js single core meme). Please tell me I am wrong
>> here, is there any other more efficient approach?
>>
>> On Sunday, July 16, 2017 at 8:06:44 PM UTC-7, peterGo wrote:
>>>
>>> Your latest benchmarks are invalid.
>>>
>>> In your benchmarks, replace b.StartTimer() with b.ResetTimer().
>>>
>>> Simply run benchmarks. Don't run race detectors. Don't run profilers.
>>> For example,
>>>
>>> $ go version
>>> go version devel +504deee Sun Jul 16 03:57:11 2017 + linux/amd64
>>> $ go test -run=! -bench=. -benchmem -cpu=1,2,4,8 pubsub_test.go
>>> goos: linux
>>> goarch: amd64
>>> BenchmarkPubSubPrimitiveChannelsMultiple2 86328
>>> ns/op  61 B/op   2 allocs/op
>>> BenchmarkPubSubPrimitiveChannelsMultiple-2  3 50844
>>> ns/op  54 B/op   2 allocs/op
>>> 

Re: [go-nuts] Re: Can't explain Golang benchmarks

[Zohaib Sibte Hassan]

Hey Jesper,

 Thanks for such an insightful reply. So the scenario I am trying to 
implement is basically a chat scenario. With huge chat rooms (thousands of 
people) situation, all the listener threads are going to run into this 
situation (which is exactly what I am doing here). To elaborate a little 
more, right now each user is a chat room is a coroutine waiting for a 
message on a channel (shameless plug project is available on 
http://raspchat.com/). Now this setup is running on a limited memory 
RaspberryPi with 512MB of RAM and few cores. What I noticed as channels 
were flooded by people on 4chan was excessive usage or CPU and memory, 
which led me to believe that channels are probably the bottle neck. So yes 
I am running it on shared memory system for now and disruptor pattern 
sounds sounds like an interesting approach I would love to try. 

Thanks for the idea, and I will follow up with my findings here

On Monday, July 17, 2017 at 10:23:04 AM UTC-7, Jesper Louis Andersen wrote:
>
> Your benchmarks are not really *doing* something once they get data. This 
> means that eventual conflicts on (internal or external) locks are going to 
> be hit all the time. In turn, you are likely to measure the performance of 
> your hardware in the event of excessive lock contention (this hypothesis 
> can be verified: block profiling).
>
> Were you to process something before going back to the barrier or channel, 
> then you are less likely to hit the problem. Though you should be aware 
> sometimes your cores will coordinate (because the work they do will tend 
> them to come back to the lock at the same point in time - a similar problem 
> to TCP incast).
>
> As you go from a single core to multiple cores, an important event happens 
> in the system. In a single-core system, you don't need the hardware to 
> coordinate. This is much faster than the case where the hardware might need 
> to tell other physical cores about what is going on. Especially under 
> contention. In short, there is a gap between the single-core system and a 
> multicore system due to communication overhead. In high-performance 
> computing a common mistake is to measure the *same* implementation in the 
> serial and parallel case, even though the serial case can be made to avoid 
> taking locks and so on. A fair benchmark would compare the fastest 
> single-core implementation with the fastest multi-core ditto.
>
> The upshot of having access to multiple cores is that you can get them all 
> to do work and this can lead to a substantial speedup. But there has to be 
> work to do. Your benchmark has almost no work to do, so most of the cores 
> are just competing against each other.
>
> For a PubSub implementation you need to think about the effort involved in 
> the implementation. Up to a point, I think a channel-based solution, or a 
> barrier based solution is adequate. For systems where you need low latency, 
> go for a pattern like the Disruptor pattern[0], which should be doable in 
> Go. This pattern works by a ring-buffer of events and two types of "clock 
> hands". One hand is the point of insertion of a new event, the other 
> hand(s) track where the readers currently are in their processing. By 
> tracking the hands through atomics, you can get messaging overhead down by 
> a lot. The trick is that each thread writes to its own hand and other 
> threads can read an eventually consistent snapshot. As long as the buffer 
> doesn't fill up completely, the eventually consistent view of an ever 
> increasing counter is enough. Getting an older value is not a big deal, as 
> long as the value isn't undefined.
>
> The problem with channels are that they are, inherently, one-shot. If a 
> message is consumed, it is gone from the channel. In a PubSub solution, you 
> really want multiple readers process the same (immutable) data and flag it 
> when it is all consumed. Message brokers and SMTP servers often optimize 
> delivery of this kind by keeping the payload separate from the header 
> (envelope): better not pay for the contents more than once. But you still 
> have to send a message to each subscriber, and this gets rather expensive. 
> The Disruptor pattern solves this.
>
> On the other hand, the Disruptor only works if you happen to have memory 
> shared. In large distributed systems, the pattern tend to break down, and 
> other methods must be used. For instance a cluster of disruptors. But then 
> keeping consistency in check becomes a challenge of the fun kind :)
>
> [0] http://lmax-exchange.github.io/disruptor/files/Disruptor-1.0.pdf - 
> Thompson, Farley, Barker, Gee, Steward, 2011
>
> On Mon, Jul 17, 2017 at 7:46 AM Zohaib Sibte Hassan  > wrote:
>
>> Awesome :) got similar results. So channels are slower for pubsub. I am 
>> surprised so far, with the efficiency going down as the cores go up (heck 
>> reminds me of that Node.js single core meme). Please tell me I am wrong 
>> here, is there any 

[go-nuts] Error: opcode not supported on this processor: mips32

[Sander van Harmelen]

Hi,

I’m trying to cross compile https://github.com/google/gousb/tree/master/lsusb 
 from macOS to mipsle using 
the following cmd:

> GOOS=linux GOARCH=mipsle CGO_ENABLED=1 CC=mipsel-openwrt-linux-gcc 
> CFLAGS=-I../include/ go build

But I receive this error:

# runtime/cgo
/usr/local/go/src/runtime/cgo/gcc_mipsx.S: Assembler messages:
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:30: Error: opcode not supported on 
this processor: mips32 (mips32) `sdc1 $f20,40($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:31: Error: opcode not supported on 
this processor: mips32 (mips32) `sdc1 $f22,48($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:32: Error: opcode not supported on 
this processor: mips32 (mips32) `sdc1 $f24,56($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:33: Error: opcode not supported on 
this processor: mips32 (mips32) `sdc1 $f26,64($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:34: Error: opcode not supported on 
this processor: mips32 (mips32) `sdc1 $f28,72($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:35: Error: opcode not supported on 
this processor: mips32 (mips32) `sdc1 $f30,80($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:52: Error: opcode not supported on 
this processor: mips32 (mips32) `ldc1 $f20,40($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:53: Error: opcode not supported on 
this processor: mips32 (mips32) `ldc1 $f22,48($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:54: Error: opcode not supported on 
this processor: mips32 (mips32) `ldc1 $f24,56($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:55: Error: opcode not supported on 
this processor: mips32 (mips32) `ldc1 $f26,64($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:56: Error: opcode not supported on 
this processor: mips32 (mips32) `ldc1 $f28,72($29)'
/usr/local/go/src/runtime/cgo/gcc_mipsx.S:57: Error: opcode not supported on 
this processor: mips32 (mips32) `ldc1 $f30,80($29)'

I’ve build mipsel-openwrt-linux-gcc using the docs written down here: 
https://wiki.openwrt.org/doc/howto/build 


Any pointers on how to make this work? Or am I out of luck and did I hit road 
block (i.e. it’s not possible/supported)?

Thanks!

Sander



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: REST API - Request Logging on Server

[Jesper Louis Andersen]

Do you need to log every request? Do you need to know the contents of the
request? Metrics are far cheaper to maintain than a log line. Also,
consider sampling, or only logging slow requests (though put a limit on
them over a period: when your system goes wrong you don't want all the slow
logging requests be the bottleneck in the system).

On Mon, Jul 17, 2017 at 7:03 PM  wrote:

> Hi Nate
>
> Thanks very much for the info
>
> Rgds,
>
> Abhi
>
>
>
> On Monday, July 17, 2017 at 10:19:58 PM UTC+5:30, Nate Finch wrote:
>>
>> Basically any log package will be pretty quick.   None of them will be
>> zero overhead, that's impossible, but most will not affect the time of your
>> requests terribly significantly.
>>
>> I personally like logrus (https://github.com/sirupsen/logrus) for
>> structured logging, it has a lot of adapters available to pipe the log
>> wherever you need it to go.  There's also a middlewear for gin called
>> ginrus that'll log to logrus.
>>
>> If you want the fastest possible logging, glog (
>> https://github.com/golang/glog) is very optimized but is just plaintext,
>> not structured, and doesn't have all the adapters of logrus.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: REST API - Request Logging on Server

[desaiabhijit]

Hi Nate

Thanks very much for the info

Rgds,

Abhi


On Monday, July 17, 2017 at 10:19:58 PM UTC+5:30, Nate Finch wrote:
>
> Basically any log package will be pretty quick.   None of them will be 
> zero overhead, that's impossible, but most will not affect the time of your 
> requests terribly significantly.   
>
> I personally like logrus (https://github.com/sirupsen/logrus) for 
> structured logging, it has a lot of adapters available to pipe the log 
> wherever you need it to go.  There's also a middlewear for gin called 
> ginrus that'll log to logrus.
>
> If you want the fastest possible logging, glog (
> https://github.com/golang/glog) is very optimized but is just plaintext, 
> not structured, and doesn't have all the adapters of logrus.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Programming paradigm.

[Tong Sun]

I feel the following is an eye-opening reading for doing functional 
programming in Go:

https://blog.gopheracademy.com/advent-2016/go-syntax-for-dsls/


On Monday, July 17, 2017 at 12:14:40 PM UTC-4, Patrick Logan wrote:
>
> I've not had any trouble doing functional programming in Go. Here's a 
> snippet of a combinator-based parseri wrote...
>
>
> https://bitbucket.org/snippets/patrickdlogan/kE9bn/fp-in-go-is-not-bad-considering-the-simple
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: REST API - Request Logging on Server

[Nate Finch]

Basically any log package will be pretty quick.   None of them will be zero 
overhead, that's impossible, but most will not affect the time of your 
requests terribly significantly.   

I personally like logrus (https://github.com/sirupsen/logrus) for 
structured logging, it has a lot of adapters available to pipe the log 
wherever you need it to go.  There's also a middlewear for gin called 
ginrus that'll log to logrus.

If you want the fastest possible logging, glog 
(https://github.com/golang/glog) is very optimized but is just plaintext, 
not structured, and doesn't have all the adapters of logrus.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Programming paradigm.

[Patrick Logan]

I've not had any trouble doing functional programming in Go. Here's a snippet 
of a combinator-based parseri wrote...

https://bitbucket.org/snippets/patrickdlogan/kE9bn/fp-in-go-is-not-bad-considering-the-simple

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[Hiroshi Ioka]

I see. Thanks.




Sent with Unibox



> On Jul 18, 2017, at 1:03 AM, Aram Hăvărneanu  wrote:
> 
> 
>> nitpick. os3_solaris.go.go set stack size by itself, instead of
>> using default one. is it intentional?
>> 
> 
> 
> 
> 
> You mean why does it explicitely ask for a 2MB stack when 2MB stacks
> are the default? I can't remember the exact reason I did that but
> it was a workaround for some sort of bug.
> 
> 
> -- 
> Aram Hăvărneanu
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[Aram Hăvărneanu]

> nitpick. os3_solaris.go.go set stack size by itself, instead of
> using default one. is it intentional?

You mean why does it explicitely ask for a 2MB stack when 2MB stacks
are the default? I can't remember the exact reason I did that but
it was a workaround for some sort of bug.

-- 
Aram Hăvărneanu

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[井岡裕史]

s/sys_solaris_amd64.go/os3_solaris.go/

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[井岡裕史]

Thank you for your detailed answer.

1. I'm convinced. I understand that I don't have to consider the difference
in general.

> On Solaris there is no difference because they create threads the
same way in all cases.

nitpick. sys_solaris_amd64.go set stack size by itself, instead of using
default one. is it intentional?

2.

> You are correct that on ELF platforms using c-shared or c-archive
> seems to force external linking, but just as I explained earlier
> about cgo, the build mode and the link mode are more or less
> conceptually orthogonal. That being said I do not know if c-shared
> and c-archive can be made to work with internal linking or not, nor
> what are the specific implications of that configuration.

c-share and c-archive enforce external linking.
https://github.com/golang/go/blob/504deee6088c2448fc3e91c94a
1ba69ec92fb7ef/src/cmd/link/internal/ld/config.go#L191-L196

I think that external linking always use runtime/cgo because of following
line.
https://github.com/golang/go/blob/504deee6088c2448fc3e91c94a
1ba69ec92fb7ef/src/cmd/link/internal/ld/lib.go#L459-L464

So, I still cannot find any possibility that newosproc0 will be called.

Hiroshi

On 17 July 2017 at 22:46, Aram Hăvărneanu  wrote:

> > 1. Is there any noticeable benchmark difference between internal
> > linking and external linking?
>
> A benchmark testing what?
>
> > IIUC, when I use external linking, runtime/cgo is used for OS thread
> > creation. In that case Go doesn't allocate the system stack itself.
>
> It's more complicated than that. By default, user-usage of cgo (when
> *you* import "C") uses external linking, stdlib-usage of cgo (e.g.
> os/user) uses internal linking, and non-cgo builds tend to use
> internal linking, however the type of linking (internal or external)
> is orthogonal to whether cgo (so runtime/cgo) is used or not.
>
> You can also use cgo with internal linking (except on systems which
> don't support it), and you can use cgo with internal linking (on
> system that support it).
>
> So your question is really about cgo, and not about linking.
>
> That being said, using runtime/cgo will indeed change the thread
> creation method on some systems, but not on others.
>
> On Solaris threads are always created the same way irrespective
> whether cgo is used or not.
>
> Plan 9 doesn't have cgo at all, so the point is moot.
>
> On Windows threads are created very similarly in both cgo and non-cgo
> cases, but there is a slight difference.
>
> On these three platforms the kernel always provides the system stack
> even when internal linking is used.
>
> On other systems like Linux and FreeBSD there indeed are fundemental
> differences to how threads are created and how stacks are allocated
> in the cgo and non-cgo cases. As you've surely discovered, on these
> other systems (most systems, really) Go allocates the system stacks
> before creating the threads, which are instructed to use this
> already-created stack.
>
> > So, I assume there are some difference related to memory usage.
>
> On Solaris there is no difference because they create threads the
> same way in all cases.
>
> On Windows, when cgo is used, the default thread stack size of 2MB
> is used for the system stack. When cgo is not used the threads are
> created with a 128kB stack instead.
>
> On other systems like Linux and FreeBSD some multiple of 8kB (usually
> 1) is used for the system stacks in the non-cgo case. In the cgo
> case, the default thread stack size is used (probably 2MB). I am
> ignoring non-standard build modes like c-shared for this discussion.
> The exact details vary in those cases but its the same fundamental
> idea.
>
> However one must note that this is virtual memory, not physical
> memory. These stacks only use address space, they only use as much
> physical memory as needed, usually one or two pages of memory (a
> page is usually 4kB but different from system to system), possibly
> with the exception of Windows which might require the use of use
> more physical memory.
>
> In principle, if you explicitely use cgo (make actual use of it)
> you might cause more physical memory to be mapped onto your stacks,
> than if you don't explicitely use cgo, but if you compare a pure
> Go program that uses runtime/cgo, with the same one that doesn't
> (perhaps due to changing linking method), there shouldn't be any
> difference in physical memory usage to a first approximation
>
> > Is runtime.newosproc0 reachable?  IIUC, newosproc0 is called by
> > _rt0_$GOARCH_$GOOS_lib if runtime/cgo is not used.  However,
> > _rt0_$GOARCH_$GOOS_lib is invoked only if buildmode=c-shared or
> > buildmode=c-archive. In addition, c-shared and c-archive always use
> > external linking. When is newosproc0 actually called?
>
> You are correct that on ELF platforms using c-shared or c-archive
> seems to force external linking, but just as I explained earlier
> about cgo, the build mode and the link mode are more or less
> conceptually 

[go-nuts] REST API - Request Logging on Server

[desaiabhijit]

Wanted to log request for REST Api on Linux with 0 ms delay on serving REST 
output

REST Api server - Gin
Reverse proxy - Varnish ( which not allow to log request )

Please anyone suggest any library

Thanks in advance

Rgds,

Abhi

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] golang and openembedded integration

[Lee Nipper]

I am adding some go components to an openembedded / yocto based project.
My question is about a reasonable approach for go building and integration 
with oe.

I am aware of the work done already to address golang within oe, 
specifically https://github.com/madisongh/meta-golang/tree/master.
This work looks very mature, robust, and well crafted.  Though I think the 
need to use recipe DEPENDS for go dependencies seems a bit redundant, but I 
understand why that is the case.

The go tool provides a fast build capability and seems sufficient during 
development, particularly with the simplicity of using GOARCH for targeting 
supported architectures.
This is in high contrast to an oe build (bitbake), which is a bit 
cumbersome and slow, but does a good job for its intended purpose.

There's also the issue of vendoring, which I think can be best addressed 
outside of oe using a normal go workspace, with govendor or other support.

So, I am considering an approach which simply uses a normal host 
development go install with "go build" or "go install" during routine 
development of go components, and avoids full integration with oe that 
would use bitbake to initiate building go components.  One integrating oe 
recipe could be used to meld the binary programs generated from a go build 
into an image for deployment. With this approach, the speed and simplicity 
of the go tooling are utilized.  The build would have two steps. Build the 
go components, then bitbake the image pulling in the go binary programs 
already built.

Have others taken any similar approach for oe and golang build environments 
?
Any comments about related oe and go integration ?

TIA,

Lee


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] slow Mod function in math/big, a better way?

[Jan Mercl]

On Mon, Jul 17, 2017 at 4:09 PM Jeff Templon 
wrote:

> Generally I'm factoring numbers that I can do in less than a day, so in
the order of 60 to 100 decimal digits.

Just for fun: Please provide a sample in that range.

-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] slow Mod function in math/big, a better way?

[Jeff Templon]

Hi

Generally I'm factoring numbers that I can do in less than a day, so in the 
order of 60 to 100 decimal digits.  

JT

On Monday, July 17, 2017 at 4:02:29 PM UTC+2, Jan Mercl wrote:
>
> On Mon, Jul 17, 2017 at 3:57 PM Jeff Templon  > wrote:
>
> > Coming back to this, now that my Go knowledge is increasing:
>
> Out of curiosity, how big are the numbers you want to factorize?
> -- 
>
> -j
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: [Bug Report] Cannot run GO in Windows Server 2016

[Egon]

Have you tried go1.9beta2? Maybe it's already fixed?

https://golang.org/dl/#unstable

If not then make a bug report to: https://github.com/golang/go/issues

+ Egon

On Monday, 17 July 2017 06:27:38 UTC-7, yanfeizh...@gmail.com wrote:
>
> [Issue]
> When running go, a problem is encountered
> 
> C:\Users\Administrator>go
> Exception 0xc005 0x8 0x0 0x0
> PC=0x0
>
> runtime.asmstdcall(0x8fdd8, 0x40ee8d, 0x10f4000, 0x0, 0x10044f980, 
> 0x20, 0x11, 0x10f4000, 0x44f8c0, 0x10f, ...)
> c:/go/src/runtime/sys_windows_amd64.s:60 +0x5e fp=0x8fd80 
> sp=0x8fd70
> rax 0x0
> rbx 0xa992c0
> rcx 0xab4e20
> rdi 0x204000
> rsi 0x8fea0
> rbp 0x8fe68
> rsp 0x8fd68
> r8  0x42e36e
> r9  0x8fee0
> r10 0xc04200
> r11 0x
> r12 0x0
> r13 0xffee
> r14 0x0
> r15 0x0
> rip 0x0
> rflags  0x10293
> cs  0x33
> fs  0x53
> gs  0x2b
> 
>
> [Environment]
>
> Version:
> Go 1.8.3 From go1.8.3.windows-amd64.msi (newest)
>
> OS:
> Microsoft Windows Server 2016 Datacenter
>
> Hardware:
> ASUSTek COMPUTER INC. Z10PA-D8 Series
> Intel(R) Xeon(R) CPU E5-2620 v4 @2.10GHz, Intel(R) Xeon(R) CPU E5-2620 v4 
> @2.10GHz
> 128GB RAM
> 512GB SSD + 2TB HHD + 6TB HHD
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] slow Mod function in math/big, a better way?

[Jan Mercl]

On Mon, Jul 17, 2017 at 3:57 PM Jeff Templon 
wrote:

> Coming back to this, now that my Go knowledge is increasing:

Out of curiosity, how big are the numbers you want to factorize?
-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] slow Mod function in math/big, a better way?

[Jeff Templon]

Coming back to this, now that my Go knowledge is increasing:

On Saturday, July 8, 2017 at 5:58:59 PM UTC+2, Andy Balholm wrote:
>
> I noticed your “naive explanation” after I sent my message. But I think it 
> is the real explanation. 
>

it turns out that 77% of the time of the program is spent in 
runtime.mallocgc :-)  I think the reason why is stuff like this:

func (z nat) shr(x nat, s uint) nat {   m := len(x) n := m - int(s/_W)  
if n <= 0 { return z[:0]}   // n > 0z = z.make(n)   
shrVU(z, x[m-n:], s%_W) return z.norm() 

  }
 
If I understand it right, there are two "make slice" operations in this 
code, and this bit is being run billions of times (heart of a tight loop). 
 Given the way Go works, the GC is unavoidable.  I've managed to make the 
code slightly faster by both swapping right-shift in instead of a Mod 
(Barrett reduction) and also avoiding my own object creation inside the 
loop, and there is 10 to 20% to be won by playing with GOGC, but the only 
way to do better is probably to rewrite the math/big library.  I tried 
using GMP but given all the temporaries created, it was even worse than 
using math/big!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] few questions about runtime implementation

[Aram Hăvărneanu]

> 1. Is there any noticeable benchmark difference between internal
> linking and external linking?

A benchmark testing what?

> IIUC, when I use external linking, runtime/cgo is used for OS thread
> creation. In that case Go doesn't allocate the system stack itself.

It's more complicated than that. By default, user-usage of cgo (when
*you* import "C") uses external linking, stdlib-usage of cgo (e.g.
os/user) uses internal linking, and non-cgo builds tend to use
internal linking, however the type of linking (internal or external)
is orthogonal to whether cgo (so runtime/cgo) is used or not.

You can also use cgo with internal linking (except on systems which
don't support it), and you can use cgo with internal linking (on
system that support it).

So your question is really about cgo, and not about linking.

That being said, using runtime/cgo will indeed change the thread
creation method on some systems, but not on others.

On Solaris threads are always created the same way irrespective
whether cgo is used or not.

Plan 9 doesn't have cgo at all, so the point is moot.

On Windows threads are created very similarly in both cgo and non-cgo
cases, but there is a slight difference.

On these three platforms the kernel always provides the system stack
even when internal linking is used.

On other systems like Linux and FreeBSD there indeed are fundemental
differences to how threads are created and how stacks are allocated
in the cgo and non-cgo cases. As you've surely discovered, on these
other systems (most systems, really) Go allocates the system stacks
before creating the threads, which are instructed to use this
already-created stack.

> So, I assume there are some difference related to memory usage.

On Solaris there is no difference because they create threads the
same way in all cases.

On Windows, when cgo is used, the default thread stack size of 2MB
is used for the system stack. When cgo is not used the threads are
created with a 128kB stack instead.

On other systems like Linux and FreeBSD some multiple of 8kB (usually
1) is used for the system stacks in the non-cgo case. In the cgo
case, the default thread stack size is used (probably 2MB). I am
ignoring non-standard build modes like c-shared for this discussion.
The exact details vary in those cases but its the same fundamental
idea.

However one must note that this is virtual memory, not physical
memory. These stacks only use address space, they only use as much
physical memory as needed, usually one or two pages of memory (a
page is usually 4kB but different from system to system), possibly
with the exception of Windows which might require the use of use
more physical memory.

In principle, if you explicitely use cgo (make actual use of it)
you might cause more physical memory to be mapped onto your stacks,
than if you don't explicitely use cgo, but if you compare a pure
Go program that uses runtime/cgo, with the same one that doesn't
(perhaps due to changing linking method), there shouldn't be any
difference in physical memory usage to a first approximation

> Is runtime.newosproc0 reachable?  IIUC, newosproc0 is called by
> _rt0_$GOARCH_$GOOS_lib if runtime/cgo is not used.  However,
> _rt0_$GOARCH_$GOOS_lib is invoked only if buildmode=c-shared or
> buildmode=c-archive. In addition, c-shared and c-archive always use
> external linking. When is newosproc0 actually called?

You are correct that on ELF platforms using c-shared or c-archive
seems to force external linking, but just as I explained earlier
about cgo, the build mode and the link mode are more or less
conceptually orthogonal. That being said I do not know if c-shared
and c-archive can be made to work with internal linking or not, nor
what are the specific implications of that configuration.

On Windows c-shared and c-archive don't seem to force external
linking, but I am not sure whether internal linking is supposed to
work or not.

-- 
Aram Hăvărneanu

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: httputil.ReverseProxy + http.DefaultTransport.DialContext results in cancelled DNS requests

[emilien . kofman . sii]

It looks like I have a similar issue with just a golang service trying to 
resolve the ip of another service through kube-dns. The error shows up when 
I submit a "large" number of requests within a few seconds. Do you have any 
update on this?

Le mercredi 28 juin 2017 20:56:40 UTC+2, Luke Kysow a écrit :
>
> Seeing this issue with golang 1.8.3 in Kubernetes but without a reverse 
> proxy. Just KubeDNS -> Kubernetes service IP -> other Golang service.
>
> When compiled with go 1.7 we saw "dial tcp: no suitable address found" 
> errors and so after looking at this issue 
>  upgraded to go 1.8.3 hoping 
> it would fix it. Instead it seems to have just been replaced with this 
> error.
>
> dial tcp: lookup other-service.default.svc.cluster.local on 10.244.0.10:53
> : dial udp 10.244.0.10:53: operation was canceled
>
>
> On Tuesday, December 20, 2016 at 3:55:17 PM UTC-8, Matt Duch wrote:
>>
>> I'm running into issues when using a ReverseProxy that sits in front of a 
>> service that serves static asset files (in this case, 100's in a few 
>> seconds). Occasionally (1 out ~500 files) will error out with:
>> http: proxy error: dial tcp: lookup myapp.default.svc.cluster.local on 
>> 172.18.0.16:53: dial udp 172.18.0.16:53: operation was canceled
>> That error occurs within a second of the request being sent, so it 
>> shouldn't trigger any timeouts.
>>
>> When using a default ReverseProxy initialized using:
>> proxy := httputil.NewSingleHostReverseProxy({Scheme: scheme,Host: 
>> host})
>>
>> In a configuration like:
>> browser -> proxy -> golang http service
>>
>> Where the proxy and http service (go 1.7.4) are running in docker 
>> containers (using kubernetes), and the DNS resolver is kube-dns (kubernetes 
>> 1.4). 
>> Changing the proxy to use a non-default transport (identical to 
>> http.DefaultTransport) but changing DialContext -> Dial:
>>
>> v.Proxy.Transport = {
>>
>> Proxy: http.ProxyFromEnvironment,
>>
>> Dial: ({
>>
>> Timeout:   30 * time.Second,
>>
>> KeepAlive: 30 * time.Second,
>>
>> }).Dial,
>>
>> MaxIdleConns:  100,
>>
>> IdleConnTimeout:   90 * time.Second,
>>
>> TLSHandshakeTimeout:   10 * time.Second,
>>
>> ExpectContinueTimeout: 1 * time.Second,
>>
>> }
>>
>> solves the problem.
>>
>> There are no errors in dnsmasq and kube-dns (the containers that provide 
>> DNS information). 
>> I've stress tested the DNS setup (hundreds of thousands of requests, with 
>> one hundred concurrently) without issues (from a go binary). 
>> I've hit the backend service directly with 1000 concurrent requests, no 
>> issues (from a go binary as well). 
>> If I switch from using a hostname in NewSingleHostReverseProxy to the IP 
>> address of the backend service, the problem is also solved.
>>
>> This seems to be a unique to named hosts + DialContext + ReverseProxy. 
>> Any ideas as to what might be happening?
>>
>
> id: 7898659753248090

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] [Bug Report] Cannot run GO in Windows Server 2016

[yanfeizhang2016a2]

[Issue]
When running go, a problem is encountered

C:\Users\Administrator>go
Exception 0xc005 0x8 0x0 0x0
PC=0x0

runtime.asmstdcall(0x8fdd8, 0x40ee8d, 0x10f4000, 0x0, 0x10044f980, 
0x20, 0x11, 0x10f4000, 0x44f8c0, 0x10f, ...)
c:/go/src/runtime/sys_windows_amd64.s:60 +0x5e fp=0x8fd80 sp=0x8fd70
rax 0x0
rbx 0xa992c0
rcx 0xab4e20
rdi 0x204000
rsi 0x8fea0
rbp 0x8fe68
rsp 0x8fd68
r8  0x42e36e
r9  0x8fee0
r10 0xc04200
r11 0x
r12 0x0
r13 0xffee
r14 0x0
r15 0x0
rip 0x0
rflags  0x10293
cs  0x33
fs  0x53
gs  0x2b


[Environment]

Version:
Go 1.8.3 From go1.8.3.windows-amd64.msi (newest)

OS:
Microsoft Windows Server 2016 Datacenter

Hardware:
ASUSTek COMPUTER INC. Z10PA-D8 Series
Intel(R) Xeon(R) CPU E5-2620 v4 @2.10GHz, Intel(R) Xeon(R) CPU E5-2620 v4 
@2.10GHz
128GB RAM
512GB SSD + 2TB HHD + 6TB HHD

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Programming paradigm.

[Egon]

I suggest reading: 
https://medium.com/@egonelbre/paradigm-is-not-the-implementation-af4c1489c073

+ Egon

On Sunday, 16 July 2017 20:42:34 UTC-7, Mayank Acharya wrote:
>
> Dear all,
> I am new to Go language.
>
> I searched a lot to understand it's paradigm structure but still not 
> getting clear idea about that.
>
> Please if anyone can explain, go language follows which programming 
> paradigm with some clear example for understanding.
>
> I really appreciate responses.
>
> Thank you.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Named or unnamed struct initialisation? What is best practice?

[roger peppe]

Sometimes I'll use non-named-member initialisation for
small structs with a well known set of fields (for example,
a Point with X and Y coordinates), but otherwise I much
prefer named initialisation (go vet will warn if you don't, in fact).
I'll almost always put each field onto its own line too - I think
it's more readable that way.

For example:

a2 := A{
name: "Mueller",
firstName: "Max",
}

It's easier to add new members that way, and the
line is shorter.

  cheers,
rog.

On 5 July 2017 at 06:17, snmed  wrote:

> Hi gophers
>
> I've a litte question about best practice for struct initialisation.  For
> example one can initialise structs like:
> type A struct {
>name  string
>firstName string
> }
>
> a1 := A{"Mueller", "Max"}
> a2 := A{name: "Mueller", firstName: "Max"}
>
> The first one is concise and intuitive, but the second is more robust in
> case of refactoring.
> If anyone adds a string field at the beginning of the struct, it compiles
> fine but changes the meaning of all initialised structs in the project, if
> nobody
> pays attention to that.
>
> Is there any best practice or advises how structs should be initialised
> and/or refactored?
>
> Cheers
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] An idea about motivating the community

[Nate Finch]

I wrote a tweet after Gophercon about making a resolution to write more 
blog posts.  I used the hashtag #GopherResolution with the hope that other 
people might pick up the idea and run with it.  So far, however, only one 
other person has used the tag.  

There are studies on the effects of making public promises (resolutions) to 
act that indicate it is generally effective in getting you to stick to 
those resolutions (at least more so than ones you don't announce).

However, it occurred to me that most resolutions fail because they're too 
vague.  There's an acronym called SMART that says your goals should be 
Specific, Measurable, Achievable, Realistic, and Time-based.  The 
"measurable" one is the interesting one, and where I think we can leverage 
software to help us.  

My idea is to make a website where gophers can make specific resolutions to 
contribute to the community, and the website can then track those 
resolutions over time.  For example, a resolution to write blog posts could 
be tracked by consuming an RSS Feed.  A resolution to contribute to a 
particular project could track PRs, issues, etc.  The website could track 
streaks over time if we give a time box for each contribution (write a blog 
post every 2 weeks, or contribute to one open source project every month).

With dashboards showing top contributors, etc, it could effectively 
game-ify community involvement.

Of course, all of this would take a significant amount of work to code, but 
it seems very doable, if enough people would actually participate.

Thoughts?  Is this something you'd be interested in?  Do you think it would 
be valuable to build?

-Nate

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Programming paradigm.

[Henry]

Go is an imperative programming language, so it will be difficult to do any 
functional programming in Go. However, beyond that, Go does not lock you into 
any particular programming paradigm. You are free to choose anything from 
procedural to object oriented or anything in between and beyond. Go is created 
to be your tools and not your religion. So just use whatever gets your jobs 
done. Stay practical and value your time.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] few questions about runtime implementation

[hirochachacha]

Hi, I have few questions out of curiosity.

1. Is there any noticeable benchmark difference between internal linking 
and external linking?
  IIUC, when I use external linking, runtime/cgo is used for OS thread 
creation. In that case Go doesn't allocate the system stack itself.
  So, I assume there are some difference related to memory usage. Can I see 
some benchmark result if any?

2. Is runtime.newosproc0 reachable?
  IIUC, newosproc0 is called by _rt0_$GOARCH_$GOOS_lib if runtime/cgo is 
not used.
  However, _rt0_$GOARCH_$GOOS_lib is invoked only if buildmode=c-shared or 
buildmode=c-archive.
  In addition, c-shared and c-archive always use external linking. When is 
newosproc0 actually called?

Thanks,

Hiroshi

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] "interface{} says nothing", lets consider it destroys information

[mhhcbon]

in func do(i interface{}) interface{} (return i), do says nothing because 
"interface{} says nothing", 

from the caller pov, it looses information at the return call, 

var x = "o"
do(x) // <- return lost the track it was a string

if you delegate that func to another func, 
it can t says anything else rather than nothing unless it introduces type 
assertion.
func to(do func(interface{})interface{}) func (interface{}) interface{} { 
return func (i interface{}) interface{} {return do(i)} }

if the observation become "interface{} destroys information", 
does it make sense to consider a "value type of any type that carries out 
its input type" ?
func do(any )  {return any}
do("thing") <- this is explicitly string

Acting the same way as interface, except that little thing about 
destroying/preserving an information.

It can be delegated to func that works on anything, still returns concrete 
types.
func to(do func(**)) func () ** { return func (i )  
{return do(i)} }

to(do)("whatever") // got a string, right ?

One step forward, 
what if  might be able to say "any type with a constraint on that 
interface",
func do(any )  {return any}
do(WhateverStringerCapable{}) <- much like using a regular parameter of 
type Stringer/interface{}, but now the return call reflects the invoke call 
, so its more complete than interface{}/Stringer.

no?

Or maybe there is a reason in destroying that information ?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.