Re: [go-nuts] Having difficulty testing this "cleanly"

[Egon]

On Wednesday, 14 September 2016 00:58:39 UTC+3, Evan Digby wrote:
>
> Hi Egon,
>
> Thanks for that. It seems to implement the same requirements as 
> implemented in my example, although I prefer my implementation as it 
> doesn't require a sleep/loop. 
>

I used sleep/loop because I probably was thinking about sleep too much at 
that time :D. It could have been done with a WaitGroup.

Anyways, coming back to the problem at hand... 
As I said there are two properties that a good concurrency primitive should 
have:

1. It should be easy to understand.
2. It should have a formal or an informal proof.

Tests are less useful, unless they are supported by the scheduler. If they 
are not, then you pretty much either end up with quite tests that don't 
cover the "hard stuff" -- e.g. the problem in your initial code.

Now you can write the tests that implement the program state-machine... 
crude version https://play.golang.org/p/wBTbA2bQTP ... then write your 
whole program in terms of Routine -- where each func is an "atomic" 
instruction... then write the Step function that it goes through all 
possible orderings... or alternatively use randomized testing and run 
through only N different orderings... Put as you can imagine it's pretty 
annoying. *(Just a top of the head idea.)*

Now there are "testing frameworks" that can help here as well... 
e.g. https://www.youtube.com/watch?v=zi0rHwfiX1Q ... 
There's https://golang.org/pkg/testing/quick/ that provides some support 
for it... i.e. you need to add an "Action" generator and a "State" 
primitive... and each call to Check does a random action and verifies 
whether "State" is in a consistent state. *(You may need to add an 
additional func "WouldWait" to do it.)*

For actually verifying/proving -- people involved with Raft have came up 
with easy-to-use framework, relatively speaking... http://verdi.uwplse.org/ 
... to say the least, proper verification requires significant effort.

Here's the simplest I was able to make the needed 
primitive https://play.golang.org/p/MkmarqPzr1 ... and instead of testing 
"Do" and "Finish" you test the rest of the interface -- they are 
deterministic -- and non-racy. (Using testing/quick here would be probably 
ideal.)

+ Egon


> The implementation I provided in the original email, with the exception of 
> the bug that Augusto pointed out, works as desired. The question was not 
> how to implement these requirements, but how to validate them with a test 
> cleanly. 
>
> Unfortunately it doesn't seem that I was clear in my original post since 
> most people are focusing on the implementation of the requirements.
>
> Thanks again for your effort in this--apologies for miscommunication the 
> original question.
>
> Evan 
>
>
> On Tue, 13 Sep 2016 at 14:48 Egon  wrote:
>
>>
>>
>> On Wednesday, 14 September 2016 00:18:26 UTC+3, Evan Digby wrote:
>>>
>>> Hi Egon,
>>>
>>> My requirements are more simple than a graceful http shutdown. I simply 
>>> require that everything that enters the RLock completes to RUnlock. 
>>> Accepted requests, or even calls to servehttp can die without issue as long 
>>> as they haven't entered the processing in the RLock.
>>>
>>
>> In that case https://play.golang.org/p/RiFEbQvytP. *Ps. Still tired, may 
>> contain bugs.*
>>
>>
>>> If the server crashes we have ways to deal with that, but it's a more 
>>> DevOps-y process. Recovering from logging, etc.,and should be an edge case 
>>> that I'm not worried about handling in code. 
>>>
>>> If the handler gets stuck in a loop, we will see that in our logging. I 
>>> don't want the server to die in that case. I want it to keep retrying (we 
>>> have exponential backoff) and informing us via structured logging of what's 
>>> going on. If it's an unanticipated loop/block, then there will be a manual 
>>> investigation into the server's state before we manually kill the process. 
>>> At that point it becomes similar to the last point, except easier because 
>>> we already know the state the message was in.
>>>
>>> In our use case we will always exit shortly after a close. It's safe to 
>>> assume the process will die after close returns.
>>>
>>> Thanks again,
>>>
>>> Evan
>>>
>>> On Tue, 13 Sep 2016 at 14:08 Egon  wrote:
>>>
>> On Tuesday, 13 September 2016 23:31:55 UTC+3, Evan Digby wrote:
>
> Hi John/Egon/Augusto,
>
> I should point out is that all we need to guarantee (barring abnormal 
> termination of course) is that once a task starts processing, it 
> finishes. 
> Partially processed messages are bad, but http requests that don't result 
> in a message being processed at all are okay.
>
> We don't need to guarantee that the result of every Accept in the HTTP 
> server results in a processed message. We handle this on the OPS side by 
> ensuring we stop sending requests to that instance before terminating the 
> process. We just want to make sure, at that point, that 

[go-nuts] Re: Max age for TCP connection

[Johnny Luo]

Hi David, 

   I saw similar solutions using DNS to add / remove node  from cluster, 
 the client side spin up a go routine keep resolving the DNS in an 
interval, if it find new ip appears in DNS record, add it to the pool , 
when ip address disappear from the DNS, it removes from the pool.
   
   and it seems to me that DNS is cached in os level.

On Wednesday, September 14, 2016 at 12:57:45 AM UTC+10, david...@ft.com 
wrote:
>
> Hi,
>
> We've got a HTTP client connecting to an active-passive cluster setup, 
> controlled through a traffic managed DNS entry and a good-to-go signal on 
> each cluster.  When the good-to-go fails, the DNS switches to point at the 
> passive cluster.  The client in question has no knowledge of whether a 
> failover has occurred, it's just hitting an endpoint on the traffic managed 
> address.
>
> The client is resolving the DNS correctly and opening a TCP connection. 
>  However, when the failover occurs, the TCP connection to the old cluster 
> stays active because of Go's persistent connections.  The connection is not 
> idle (it still makes scheduled requests to the endpoint via its IP address) 
> and so is not closed when we close all idle connections.
>
> Is there some way to set a maximum age for the TCP connection?
>
> --
>
> *This email was sent by a company owned by Financial Times Group Limited 
> ("FT Group "), 
> registered office at Number One Southwark Bridge, London SE1 9HL.  
> Registered in England and Wales with company number 879531. This e-mail may 
> contain confidential information. If you are not the intended recipient, 
> please notify the sender immediately, delete all copies and do not 
> distribute it further.  It could also contain personal views which are not 
> necessarily those of the FT Group.  We may monitor outgoing or 
> incoming emails as permitted by law.*
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] handling duplicate keys in yaml

[Joseph Lorenzini]

Hi all,

I am using go-yaml to parse yaml files. I noticed that if there are 
duplicate keys in a yaml map, then the parser silently replaces the 
previous value with the new one. Is there any way to treat this as a parser 
exception with this library or would I need to create a custom parser for 
that?

Thanks,
Joe 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Having difficulty testing this "cleanly"

[aroman]

By the way, the latest code at https://play.golang.org/p/QTkSJAOPtE isn't 
great, because once Close() is called, new tasks will be blocked until 
existing tasks finish.

That is, once Lock() is called in Close(), subsequent RLocks() are blocked. 
 At least, this is document in an old version of the docs (1.5.2):

> Lock locks rw for writing. If the lock is already locked for reading or 
> writing, Lock blocks until the lock is available. To ensure that the lock 
> eventually becomes available, a blocked Lock call excludes new readers from 
> acquiring the lock.


However, current docs don't indicate 
that: https://godoc.org/sync#RWMutex.Lock
But the code shows that it does: 
https://github.com/golang/go/blob/master/src/sync/rwmutex.go#L80

So if the processing takes a long time, many goroutines may be stuck 
waiting for it to resolve before being rejected.

- Augusto

On Tuesday, September 13, 2016 at 5:09:59 PM UTC-7, aro...@gmail.com wrote:
>
> The way to test it is something like this:  
> https://play.golang.org/p/8hN9q9ibIc
>
> Specifically, have the handler push to start & finish channels. In the 
> monitoring goroutine, you can record events for start, finish, & close (and 
> anything else you want to check, maybe Queue?).  After the call to Close() 
> completes, you can check through the event history.  Depending on how you 
> set up your test, you could see, for example, that nothing starts after the 
> CLOSED event.
>
> The test that I made is a little weak in that it tries to verify that 
> exactly N are started which means that to avoid races I don't try to submit 
> to-be-rejected requests until after the channel is closed.  Perhaps it 
> would be better to also have a separate tests that continuously submits 
> requests (e.g. until 100 have failed) and then check the event log to make 
> sure that none start after close and that all started before close are 
> finished before close returns.
>
> I think that satisfies your testing criteria, although it doesn't 100% 
> verify that everything is finished before Close returns -- it's possible 
> that you get lucky and the entire machine freezes on return from Close() 
> and then the queued goroutines flush before the call to read the event log. 
>  If you really want to make that testable, you can put a hook in Close() 
> that sends a signal before it returns.
>
> - Augusto
>
> On Tuesday, September 13, 2016 at 2:06:43 PM UTC-7, Evan Digby wrote:
>>
>> Hi John,
>>
>> Thank you!
>>
>> The h.closed channel, if checked properly (after the RLock), prevents the 
>> race condition that Augusto pointed out in his post a few back.
>>
>> I fixed my implementation here: https://play.golang.org/p/QTkSJAOPtE
>>
>> Thanks again,
>>
>> Evan
>>
>> On Tue, 13 Sep 2016 at 14:04 John Souvestre  wrote:
>>
>>> OK.  Give me a minute to add that.  I just wanted to make sure I was 
>>> headed in the right direction.  J
>>>
>>>  
>>>
>>> Note:  In looking at your original code I didn’t see any way that the 
>>> error could happen, so I ignored that case.  Given this, there was no need 
>>> for the h.closed channel.
>>>
>>>  
>>>
>>> Back in a few.  J
>>>
>>>  
>>>
>>> John
>>>
>>> John Souvestre - New Orleans LA
>>>
>>>  
>>>
>>> *From:* Evan Digby [mailto:evan...@gmail.com] 
>>> *Sent:* 2016 September 13, Tue 15:59
>>> *To:* John Souvestre; golang-nuts
>>>
>>>
>>> *Subject:* Re: [go-nuts] Having difficulty testing this "cleanly"
>>>
>>>  
>>>
>>> Hi John,
>>>
>>>  
>>>
>>> What you've posted is a valid way to implement the handler, but not a 
>>> way to validate it.
>>>
>>>  
>>>
>>> The implementation in the example isn't the problem. It's how to 
>>> validate the implementation with a test.
>>>
>>>  
>>>
>>> If we add a WaitGroup.Wait inside the handler then the test is not valid 
>>> because it will wait until they're done. If the test does the waiting, then 
>>> we aren't validating that the implementation itself does the waiting. 
>>>
>>>  
>>>
>>> I'm trying to find a clean way to validate that the waiting is done by 
>>> the Close call.
>>>
>>>  
>>>
>>> Thanks again for your effort in this!
>>>
>>>  
>>>
>>> Evan
>>>
>>>  
>>>
>>> On Tue, 13 Sep 2016 at 13:52 John Souvestre  wrote:
>>>
>>> Hi Evan.
>>>
>>>  
>>>
>>> I still don’t quite understand exactly what you are shooting for.  I 
>>> tried to reimplement what you posted originally.  Check out 
>>> https://play.golang.org/p/koUJYCKFpa.  Does this come close 
>>> functionally?
>>>
>>>  
>>>
>>> John
>>>
>>> John Souvestre - New Orleans LA
>>>
>>>  
>>>
>>> *From:* golan...@googlegroups.com [mailto:golan...@googlegroups.com] *On 
>>> Behalf Of *Evan Digby
>>> *Sent:* 2016 September 13, Tue 15:32
>>> *To:* golang-nuts
>>> *Cc:* aro...@gmail.com
>>> *Subject:* Re: [go-nuts] Having difficulty testing this "cleanly"
>>>
>>>  
>>>
>>> Hi John/Egon/Augusto,
>>>
>>>  
>>>
>>> I should point out is that all we need to guarantee (barring abnormal 
>>> termination of 

Re: [go-nuts] Having difficulty testing this "cleanly"

[aroman]

The way to test it is something like this: 
 https://play.golang.org/p/8hN9q9ibIc

Specifically, have the handler push to start & finish channels. In the 
monitoring goroutine, you can record events for start, finish, & close (and 
anything else you want to check, maybe Queue?).  After the call to Close() 
completes, you can check through the event history.  Depending on how you 
set up your test, you could see, for example, that nothing starts after the 
CLOSED event.

The test that I made is a little weak in that it tries to verify that 
exactly N are started which means that to avoid races I don't try to submit 
to-be-rejected requests until after the channel is closed.  Perhaps it 
would be better to also have a separate tests that continuously submits 
requests (e.g. until 100 have failed) and then check the event log to make 
sure that none start after close and that all started before close are 
finished before close returns.

I think that satisfies your testing criteria, although it doesn't 100% 
verify that everything is finished before Close returns -- it's possible 
that you get lucky and the entire machine freezes on return from Close() 
and then the queued goroutines flush before the call to read the event log. 
 If you really want to make that testable, you can put a hook in Close() 
that sends a signal before it returns.

- Augusto

On Tuesday, September 13, 2016 at 2:06:43 PM UTC-7, Evan Digby wrote:
>
> Hi John,
>
> Thank you!
>
> The h.closed channel, if checked properly (after the RLock), prevents the 
> race condition that Augusto pointed out in his post a few back.
>
> I fixed my implementation here: https://play.golang.org/p/QTkSJAOPtE
>
> Thanks again,
>
> Evan
>
> On Tue, 13 Sep 2016 at 14:04 John Souvestre  > wrote:
>
>> OK.  Give me a minute to add that.  I just wanted to make sure I was 
>> headed in the right direction.  J
>>
>>  
>>
>> Note:  In looking at your original code I didn’t see any way that the 
>> error could happen, so I ignored that case.  Given this, there was no need 
>> for the h.closed channel.
>>
>>  
>>
>> Back in a few.  J
>>
>>  
>>
>> John
>>
>> John Souvestre - New Orleans LA
>>
>>  
>>
>> *From:* Evan Digby [mailto:evan...@gmail.com ] 
>> *Sent:* 2016 September 13, Tue 15:59
>> *To:* John Souvestre; golang-nuts
>>
>>
>> *Subject:* Re: [go-nuts] Having difficulty testing this "cleanly"
>>
>>  
>>
>> Hi John,
>>
>>  
>>
>> What you've posted is a valid way to implement the handler, but not a way 
>> to validate it.
>>
>>  
>>
>> The implementation in the example isn't the problem. It's how to validate 
>> the implementation with a test.
>>
>>  
>>
>> If we add a WaitGroup.Wait inside the handler then the test is not valid 
>> because it will wait until they're done. If the test does the waiting, then 
>> we aren't validating that the implementation itself does the waiting. 
>>
>>  
>>
>> I'm trying to find a clean way to validate that the waiting is done by 
>> the Close call.
>>
>>  
>>
>> Thanks again for your effort in this!
>>
>>  
>>
>> Evan
>>
>>  
>>
>> On Tue, 13 Sep 2016 at 13:52 John Souvestre > > wrote:
>>
>> Hi Evan.
>>
>>  
>>
>> I still don’t quite understand exactly what you are shooting for.  I 
>> tried to reimplement what you posted originally.  Check out 
>> https://play.golang.org/p/koUJYCKFpa.  Does this come close functionally?
>>
>>  
>>
>> John
>>
>> John Souvestre - New Orleans LA
>>
>>  
>>
>> *From:* golan...@googlegroups.com  [mailto:
>> golan...@googlegroups.com ] *On Behalf Of *Evan Digby
>> *Sent:* 2016 September 13, Tue 15:32
>> *To:* golang-nuts
>> *Cc:* aro...@gmail.com 
>> *Subject:* Re: [go-nuts] Having difficulty testing this "cleanly"
>>
>>  
>>
>> Hi John/Egon/Augusto,
>>
>>  
>>
>> I should point out is that all we need to guarantee (barring abnormal 
>> termination of course) is that once a task starts processing, it finishes. 
>> Partially processed messages are bad, but http requests that don't result 
>> in a message being processed at all are okay.
>>
>>  
>>
>> We don't need to guarantee that the result of every Accept in the HTTP 
>> server results in a processed message. We handle this on the OPS side by 
>> ensuring we stop sending requests to that instance before terminating the 
>> process. We just want to make sure, at that point, that the messages which 
>> did make it to the handler are flushed.
>>
>>  
>>
>> So the case where:
>>
>>  
>>
>> h.Handle(...)  <-- gets past the closed channel check, calls go ..., butthe 
>> goroutine doesn't execute yet.
>> h.Close() <-- closes the close channel, Locks and Unlocks,returns.
>> ...now the goroutine executes and acquires the read lock.
>>
>>  
>>
>> We actually don't care if "Handle" completes in this example. We only 
>> care if that our task handler starts processing a message that it completes 
>> the processing.
>>
>>  
>>
>> Thanks again,
>>
>> Evan
>>
>>
>> On Tuesday, 13 September 2016 13:24:03 

Re: [go-nuts] Cross-compilation of golang assembly via 'go tool asm'

[pcj127]

Thanks, great reference to gooOsArchFile()

On Tuesday, September 13, 2016 at 11:04:44 AM UTC-6, ironi...@gmail.com 
wrote:
>
>
>
> On Tuesday, September 13, 2016 at 11:08:18 AM UTC-5, pcj...@gmail.com 
> wrote:
>>
>> My next question is which tool is responsible for selecting the correct 
>> sha1block_GOARCH.s? Is the the compiler, linker, or where?
>>
>  
> See https://golang.org/pkg/go/build/ (under Build Constraints)
> And (as of Go 1.7.1) https://golang.org/src/go/build/build.go#L1449 
> goodOSArchFile()
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Having difficulty testing this "cleanly"

[Evan Digby]

Hi Egon,

Thanks for that. It seems to implement the same requirements as implemented
in my example, although I prefer my implementation as it doesn't require a
sleep/loop.

The implementation I provided in the original email, with the exception of
the bug that Augusto pointed out, works as desired. The question was not
how to implement these requirements, but how to validate them with a test
cleanly.

Unfortunately it doesn't seem that I was clear in my original post since
most people are focusing on the implementation of the requirements.

Thanks again for your effort in this--apologies for miscommunication the
original question.

Evan


On Tue, 13 Sep 2016 at 14:48 Egon  wrote:

>
>
> On Wednesday, 14 September 2016 00:18:26 UTC+3, Evan Digby wrote:
>>
>> Hi Egon,
>>
>> My requirements are more simple than a graceful http shutdown. I simply
>> require that everything that enters the RLock completes to RUnlock.
>> Accepted requests, or even calls to servehttp can die without issue as long
>> as they haven't entered the processing in the RLock.
>>
>
> In that case https://play.golang.org/p/RiFEbQvytP. *Ps. Still tired, may
> contain bugs.*
>
>
>> If the server crashes we have ways to deal with that, but it's a more
>> DevOps-y process. Recovering from logging, etc.,and should be an edge case
>> that I'm not worried about handling in code.
>>
>> If the handler gets stuck in a loop, we will see that in our logging. I
>> don't want the server to die in that case. I want it to keep retrying (we
>> have exponential backoff) and informing us via structured logging of what's
>> going on. If it's an unanticipated loop/block, then there will be a manual
>> investigation into the server's state before we manually kill the process.
>> At that point it becomes similar to the last point, except easier because
>> we already know the state the message was in.
>>
>> In our use case we will always exit shortly after a close. It's safe to
>> assume the process will die after close returns.
>>
>> Thanks again,
>>
>> Evan
>>
>> On Tue, 13 Sep 2016 at 14:08 Egon  wrote:
>>
> On Tuesday, 13 September 2016 23:31:55 UTC+3, Evan Digby wrote:

 Hi John/Egon/Augusto,

 I should point out is that all we need to guarantee (barring abnormal
 termination of course) is that once a task starts processing, it finishes.
 Partially processed messages are bad, but http requests that don't result
 in a message being processed at all are okay.

 We don't need to guarantee that the result of every Accept in the HTTP
 server results in a processed message. We handle this on the OPS side by
 ensuring we stop sending requests to that instance before terminating the
 process. We just want to make sure, at that point, that the messages which
 did make it to the handler are flushed.

 So the case where:

 h.Handle(...)  <-- gets past the closed channel check, calls go ..., butthe
 goroutine doesn't execute yet.
 h.Close() <-- closes the close channel, Locks and Unlocks,returns.
 ...now the goroutine executes and acquires the read lock.

 We actually don't care if "Handle" completes in this example. We only
 care if that our task handler starts processing a message that it completes
 the processing.

>>>
>>> How do you actually ensure that it completes processing without hooking
>>> into Server? I.e. that buffers and sockets get properly flushed?
>>>
>>> Let's take a step back and what are the properties that you need --
>>>
>>> I assume it's just graceful shutdown where all the pending ServeHTTP
>>> requests have been processed?
>>>
>>> What should happen when the server crashes -- is it vital for those
>>> requests to be processed, once they have been accepted?
>>>
>>> What should happen when one handler gets stuck in an infinite wait/loop?
>>>
>>> Does the "Close" returning mean you exit main or does hpw does the
>>> process termination depend on it? Or is it just another goroutine that is
>>> terminating not the whole process?
>>>
>>> + Egon
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "golang-nuts" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/golang-nuts/jh-nvt9ukBg/unsubscribe.
>>>
>> To unsubscribe from this group and all its topics, send an email to
>>> golang-nuts...@googlegroups.com.
>>
>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/jh-nvt9ukBg/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are 

Re: [go-nuts] Re: Two Related Questions. (1) Map Concurrency (2) Making Multidimensional Arrays

[Justin Israel]

On Tue, Sep 13, 2016 at 3:31 PM  wrote:

> I've spent many, many, many hours on this (measured in days, at this
> point) and I'm convinced that Golang is missing a fundamental programming
> capability, that of multidimensional associative arrays. Searching through
> this forum's archives and other sites, I've found that I'm not the first
> one to face this problem and no one who has posted about this has yet
> received a decent response (at least, not one that I have found)! The best
> responses consist either of maps that are, at most, two levels deep or maps
> whose depth is known in advance and declared using something like
> make(map[string]map[string]map[string]map[string]map[string]string) and
> even in those cases, one cannot replace a map with just a string on any of
> those levels.
>
> This is a MAJOR missing feature in Golang.
>
> In PHP, for example, multidimensional associative arrays are declarable
> and usable in a single line of code. In Javascript, Objects can serve that
> purpose and those serve as the foundation of pretty much the entire
> language. HTML is built on the tree structure, which is essentially just a
> multimentional array. The Internet is essentiallly built on it
> multidimensional associative arrays! Take XML or JSON, both of which are
> essentially multidimensional associative arrays. Take the file/folder
> system on your desktop computers and on servers as another example. Take
> relational database recordsets. I mean, the list goes on and on and on! How
> can Golang not have a simple way to initialize, manipulate and maintain
> multidimensional associative arrays?
>
> This is such a big problem that I've even spent time looking into
> alternate options to Golang, such as Clojure, Erlang and Elixir, and I'm
> quite certain others have followed my route. I am not about to give up on
> Golang just yet, because it would mean a tremendous amount of rewriting
> what I've already done and learning yet another new language, but this is a
> very serious missing piece in Golang.
>
> It's incredible that this has gone overlooked in Golang for so long. The
> posts I found were from as early as 2012! This is a very disappointing
> discovery. I have found many things about Golang that I really like and
> many things about Golang that I really don't like, but the lack of
> multidimensional associative arrays beats out even the lack of decent
> global error handling! By the way, while I'm at it are there any libraries
> for that?
>
> Is there any of doing multidimensional associative arrays in Golang
> (beyond two levels and without specifying the depth of the array or map in
> advance) or am I wasting my time on this problem? If not, does anyone know
> if this feature is planned for an upcoming version of Golang?
>

When you say this is a missing feature of Go and that you can't imagine how
Go programmers have expressed solutions to common problems without it, I
interpret that as (and I apologize that you probably won't agree) "this
approach that I do in PHP or Javascript does work exactly the same way in
Go, so I am calling it a missing feature". The thing is, PHP and Javascript
are both dynamically typed languages. They would both allow you to shove
any types that you want, mixed, into data structures. Thus you would be
able to express your map of strings to either a string or another map. In a
statically-type language like Go, you have to either be explicit about the
concrete types you are storing, or you have to use interfaces/reflection in
order to mix types with common behaviour and then inspect and extract the
concrete type or call the interface methods.

Because I am more familiar with Python than PHP/JS, let me use that as a
comparison. In Python you can create a dictionary with any kind of hashable
key type and any value, mixed:

aDict = {
(1, 2, 'c'): "value",
'string': {'another': 'value'},
1: ['a', 'b', 3],
}

​
But really every object in Python is just a C wrapper object called a
PyObject (that is, in CPython). Python will then dynamically evaluate the
type and resolve methods and operations on them as needed. When you have a
mixed dictionary (multi-dimensional associative array) of more nested dicts
and string values, it allows you to do things like:   aDict['a']['b']['c']
because on-the-fly, Python would try and treat each value as a dict and it
would either work at runtime. or raise an exception if the object stored at
that point did not support the operation.
Go, being a statically typed language, can have the mixed types, but the
1-to-1 equivalent (or closest translation) of a "GoObject" (PyObject) would
be interfaces. And then you need to manually assert the type, whereas
Python would be doing it dynamically when you simply try to use the object,
because that is how it does it for all objects whenever you access them.

But honestly, with all the examples you listed in the world of
tree-structured data, that does not mean it 

Re: [go-nuts] Having difficulty testing this "cleanly"

[Egon]

On Tuesday, 13 September 2016 23:31:55 UTC+3, Evan Digby wrote:
>
> Hi John/Egon/Augusto,
>
> I should point out is that all we need to guarantee (barring abnormal 
> termination of course) is that once a task starts processing, it finishes. 
> Partially processed messages are bad, but http requests that don't result 
> in a message being processed at all are okay.
>
> We don't need to guarantee that the result of every Accept in the HTTP 
> server results in a processed message. We handle this on the OPS side by 
> ensuring we stop sending requests to that instance before terminating the 
> process. We just want to make sure, at that point, that the messages which 
> did make it to the handler are flushed.
>
> So the case where:
>
> h.Handle(...)  <-- gets past the closed channel check, calls go ..., butthe 
> goroutine doesn't execute yet.
> h.Close() <-- closes the close channel, Locks and Unlocks,returns.
> ...now the goroutine executes and acquires the read lock.
>
> We actually don't care if "Handle" completes in this example. We only care 
> if that our task handler starts processing a message that it completes the 
> processing.
>

How do you actually ensure that it completes processing without hooking 
into Server? I.e. that buffers and sockets get properly flushed?

Let's take a step back and what are the properties that you need -- 

I assume it's just graceful shutdown where all the pending ServeHTTP 
requests have been processed?

What should happen when the server crashes -- is it vital for those 
requests to be processed, once they have been accepted?

What should happen when one handler gets stuck in an infinite wait/loop?

Does the "Close" returning mean you exit main or does hpw does the process 
termination depend on it? Or is it just another goroutine that is 
terminating not the whole process?

+ Egon

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Having difficulty testing this "cleanly"

[Evan Digby]

Hi John,

Thank you!

The h.closed channel, if checked properly (after the RLock), prevents the
race condition that Augusto pointed out in his post a few back.

I fixed my implementation here: https://play.golang.org/p/QTkSJAOPtE

Thanks again,

Evan

On Tue, 13 Sep 2016 at 14:04 John Souvestre  wrote:

> OK.  Give me a minute to add that.  I just wanted to make sure I was
> headed in the right direction.  J
>
>
>
> Note:  In looking at your original code I didn’t see any way that the
> error could happen, so I ignored that case.  Given this, there was no need
> for the h.closed channel.
>
>
>
> Back in a few.  J
>
>
>
> John
>
> John Souvestre - New Orleans LA
>
>
>
> *From:* Evan Digby [mailto:evandi...@gmail.com]
> *Sent:* 2016 September 13, Tue 15:59
> *To:* John Souvestre; golang-nuts
>
>
> *Subject:* Re: [go-nuts] Having difficulty testing this "cleanly"
>
>
>
> Hi John,
>
>
>
> What you've posted is a valid way to implement the handler, but not a way
> to validate it.
>
>
>
> The implementation in the example isn't the problem. It's how to validate
> the implementation with a test.
>
>
>
> If we add a WaitGroup.Wait inside the handler then the test is not valid
> because it will wait until they're done. If the test does the waiting, then
> we aren't validating that the implementation itself does the waiting.
>
>
>
> I'm trying to find a clean way to validate that the waiting is done by the
> Close call.
>
>
>
> Thanks again for your effort in this!
>
>
>
> Evan
>
>
>
> On Tue, 13 Sep 2016 at 13:52 John Souvestre  wrote:
>
> Hi Evan.
>
>
>
> I still don’t quite understand exactly what you are shooting for.  I tried
> to reimplement what you posted originally.  Check out
> https://play.golang.org/p/koUJYCKFpa.  Does this come close functionally?
>
>
>
> John
>
> John Souvestre - New Orleans LA
>
>
>
> *From:* golang-nuts@googlegroups.com [mailto:golang-nuts@googlegroups.com]
> *On Behalf Of *Evan Digby
> *Sent:* 2016 September 13, Tue 15:32
> *To:* golang-nuts
> *Cc:* aro...@gmail.com
> *Subject:* Re: [go-nuts] Having difficulty testing this "cleanly"
>
>
>
> Hi John/Egon/Augusto,
>
>
>
> I should point out is that all we need to guarantee (barring abnormal
> termination of course) is that once a task starts processing, it finishes.
> Partially processed messages are bad, but http requests that don't result
> in a message being processed at all are okay.
>
>
>
> We don't need to guarantee that the result of every Accept in the HTTP
> server results in a processed message. We handle this on the OPS side by
> ensuring we stop sending requests to that instance before terminating the
> process. We just want to make sure, at that point, that the messages which
> did make it to the handler are flushed.
>
>
>
> So the case where:
>
>
>
> h.Handle(...)  <-- gets past the closed channel check, calls go ..., butthe
> goroutine doesn't execute yet.
> h.Close() <-- closes the close channel, Locks and Unlocks,returns.
> ...now the goroutine executes and acquires the read lock.
>
>
>
> We actually don't care if "Handle" completes in this example. We only care
> if that our task handler starts processing a message that it completes the
> processing.
>
>
>
> Thanks again,
>
> Evan
>
>
> On Tuesday, 13 September 2016 13:24:03 UTC-7, aro...@gmail.com wrote:
>
> The mutex approach is fundamentally broken because you can't guarantee
> that the tasks are all started (and have a read-lock acquired) before you
> call close.
>
>
>
> Consider:
>
> h.Handle(...)  <-- gets past the closed channel check, calls go ...,
> butthe goroutine doesn't execute yet.
> h.Close() <-- closes the close channel, Locks and Unlocks,returns.
> ...now the goroutine executes and acquires the read lock.
>
>
>
> So really, if you can't control the Handle() function, you need two
> WaitGroups:  one to verify that all goroutines have started before shutting
> down the task handler and a second one for all goroutines to have
> finished.  However, it's tricky if we don't know the real use case.
>
>
> Sounds like you are trying to do graceful http shutdown.  Have you looked
> at other libraries that do that?  If you don't have a way to account for
> the time between Handle(..) is called and the goroutine starts, you always
> might miss a task that got called near the time Close() was called.
>
>
>
> - Augusto
>
>
> On Tuesday, September 13, 2016 at 12:50:50 PM UTC-7, Evan Digby wrote:
>
> Hi Aroman,
>
>
>
> Your approach using the WaitGroup is definitely better in this toy
> example. The reason I didn't use the WaitGroup is because the non-toy
> example is wrapping the HTTP Server handler. I have no way to inject an
> "add" before the goroutine is created since that's handled by Go's HTTP
> Server without re-implementing the accept->handle loop using the listener.
>
>
>
> Apologies for not giving the full context in the example.
>
>
>
> I'm not sure how it could block an outstanding task since the closed
> 

RE: [go-nuts] Having difficulty testing this "cleanly"

[John Souvestre]

OK.  Give me a minute to add that.  I just wanted to make sure I was headed in 
the right direction.  J

 

Note:  In looking at your original code I didn’t see any way that the error 
could happen, so I ignored that case.  Given this, there was no need for the 
h.closed channel.

 

Back in a few.  J

 

John

John Souvestre - New Orleans LA

 

From: Evan Digby [mailto:evandi...@gmail.com] 
Sent: 2016 September 13, Tue 15:59
To: John Souvestre; golang-nuts
Subject: Re: [go-nuts] Having difficulty testing this "cleanly"

 

Hi John,

 

What you've posted is a valid way to implement the handler, but not a way to 
validate it.

 

The implementation in the example isn't the problem. It's how to validate the 
implementation with a test.

 

If we add a WaitGroup.Wait inside the handler then the test is not valid 
because it will wait until they're done. If the test does the waiting, then we 
aren't validating that the implementation itself does the waiting. 

 

I'm trying to find a clean way to validate that the waiting is done by the 
Close call.

 

Thanks again for your effort in this!

 

Evan

 

On Tue, 13 Sep 2016 at 13:52 John Souvestre  wrote:

Hi Evan.

 

I still don’t quite understand exactly what you are shooting for.  I tried to 
reimplement what you posted originally.  Check out 
https://play.golang.org/p/koUJYCKFpa.  Does this come close functionally?

 

John

John Souvestre - New Orleans LA

 

From: golang-nuts@googlegroups.com [mailto:golang-nuts@googlegroups.com] On 
Behalf Of Evan Digby
Sent: 2016 September 13, Tue 15:32
To: golang-nuts
Cc: aro...@gmail.com
Subject: Re: [go-nuts] Having difficulty testing this "cleanly"

 

Hi John/Egon/Augusto,

 

I should point out is that all we need to guarantee (barring abnormal 
termination of course) is that once a task starts processing, it finishes. 
Partially processed messages are bad, but http requests that don't result in a 
message being processed at all are okay.

 

We don't need to guarantee that the result of every Accept in the HTTP server 
results in a processed message. We handle this on the OPS side by ensuring we 
stop sending requests to that instance before terminating the process. We just 
want to make sure, at that point, that the messages which did make it to the 
handler are flushed.

 

So the case where:

 

h.Handle(...)  <-- gets past the closed channel check, calls go ..., butthe 
goroutine doesn't execute yet.
h.Close() <-- closes the close channel, Locks and Unlocks,returns.
...now the goroutine executes and acquires the read lock.

 

We actually don't care if "Handle" completes in this example. We only care if 
that our task handler starts processing a message that it completes the 
processing.

 

Thanks again,

Evan


On Tuesday, 13 September 2016 13:24:03 UTC-7, aro...@gmail.com wrote:

The mutex approach is fundamentally broken because you can't guarantee that the 
tasks are all started (and have a read-lock acquired) before you call close.

 

Consider:

h.Handle(...)  <-- gets past the closed channel check, calls go ..., butthe 
goroutine doesn't execute yet.
h.Close() <-- closes the close channel, Locks and Unlocks,returns.
...now the goroutine executes and acquires the read lock.

 

So really, if you can't control the Handle() function, you need two WaitGroups: 
 one to verify that all goroutines have started before shutting down the task 
handler and a second one for all goroutines to have finished.  However, it's 
tricky if we don't know the real use case.


Sounds like you are trying to do graceful http shutdown.  Have you looked at 
other libraries that do that?  If you don't have a way to account for the time 
between Handle(..) is called and the goroutine starts, you always might miss a 
task that got called near the time Close() was called.

 

- Augusto


On Tuesday, September 13, 2016 at 12:50:50 PM UTC-7, Evan Digby wrote:

Hi Aroman,

 

Your approach using the WaitGroup is definitely better in this toy example. The 
reason I didn't use the WaitGroup is because the non-toy example is wrapping 
the HTTP Server handler. I have no way to inject an "add" before the goroutine 
is created since that's handled by Go's HTTP Server without re-implementing the 
accept->handle loop using the listener. 

 

Apologies for not giving the full context in the example.  

 

I'm not sure how it could block an outstanding task since the closed channel is 
called before the Lock(), so no additional calls to RLock will be made at that 
point, and the Lock will just wait until all of the RLocks are complete.

 

Regarding your testing strategy, I do like it better than any of my current 
strategy; however, There still is a chance that a task could complete between 
lines 90 and 91:

 

h.Close()

events <- ALL_TASKS_FINISHED

 

So this doesn't solve the racy-ness I'm concerned about unless you put an 
arbitrary sleep in the handlers, which I'm trying to avoid. 

Re: [go-nuts] Having difficulty testing this "cleanly"

[Evan Digby]

Hi John,

What you've posted is a valid way to implement the handler, but not a way
to validate it.

The implementation in the example isn't the problem. It's how to validate
the implementation with a test.

If we add a WaitGroup.Wait inside the handler then the test is not valid
because it will wait until they're done. If the test does the waiting, then
we aren't validating that the implementation itself does the waiting.

I'm trying to find a clean way to validate that the waiting is done by the
Close call.

Thanks again for your effort in this!

Evan

On Tue, 13 Sep 2016 at 13:52 John Souvestre  wrote:

> Hi Evan.
>
>
>
> I still don’t quite understand exactly what you are shooting for.  I tried
> to reimplement what you posted originally.  Check out
> https://play.golang.org/p/koUJYCKFpa.  Does this come close functionally?
>
>
>
> John
>
> John Souvestre - New Orleans LA
>
>
>
> *From:* golang-nuts@googlegroups.com [mailto:golang-nuts@googlegroups.com]
> *On Behalf Of *Evan Digby
> *Sent:* 2016 September 13, Tue 15:32
> *To:* golang-nuts
> *Cc:* aro...@gmail.com
> *Subject:* Re: [go-nuts] Having difficulty testing this "cleanly"
>
>
>
> Hi John/Egon/Augusto,
>
>
>
> I should point out is that all we need to guarantee (barring abnormal
> termination of course) is that once a task starts processing, it finishes.
> Partially processed messages are bad, but http requests that don't result
> in a message being processed at all are okay.
>
>
>
> We don't need to guarantee that the result of every Accept in the HTTP
> server results in a processed message. We handle this on the OPS side by
> ensuring we stop sending requests to that instance before terminating the
> process. We just want to make sure, at that point, that the messages which
> did make it to the handler are flushed.
>
>
>
> So the case where:
>
>
>
> h.Handle(...)  <-- gets past the closed channel check, calls go ..., butthe
> goroutine doesn't execute yet.
> h.Close() <-- closes the close channel, Locks and Unlocks,returns.
> ...now the goroutine executes and acquires the read lock.
>
>
>
> We actually don't care if "Handle" completes in this example. We only care
> if that our task handler starts processing a message that it completes the
> processing.
>
>
>
> Thanks again,
>
> Evan
>
>
> On Tuesday, 13 September 2016 13:24:03 UTC-7, aro...@gmail.com wrote:
>
> The mutex approach is fundamentally broken because you can't guarantee
> that the tasks are all started (and have a read-lock acquired) before you
> call close.
>
>
>
> Consider:
>
> h.Handle(...)  <-- gets past the closed channel check, calls go ...,
> butthe goroutine doesn't execute yet.
> h.Close() <-- closes the close channel, Locks and Unlocks,returns.
> ...now the goroutine executes and acquires the read lock.
>
>
>
> So really, if you can't control the Handle() function, you need two
> WaitGroups:  one to verify that all goroutines have started before shutting
> down the task handler and a second one for all goroutines to have
> finished.  However, it's tricky if we don't know the real use case.
>
>
> Sounds like you are trying to do graceful http shutdown.  Have you looked
> at other libraries that do that?  If you don't have a way to account for
> the time between Handle(..) is called and the goroutine starts, you always
> might miss a task that got called near the time Close() was called.
>
>
>
> - Augusto
>
>
> On Tuesday, September 13, 2016 at 12:50:50 PM UTC-7, Evan Digby wrote:
>
> Hi Aroman,
>
>
>
> Your approach using the WaitGroup is definitely better in this toy
> example. The reason I didn't use the WaitGroup is because the non-toy
> example is wrapping the HTTP Server handler. I have no way to inject an
> "add" before the goroutine is created since that's handled by Go's HTTP
> Server without re-implementing the accept->handle loop using the listener.
>
>
>
> Apologies for not giving the full context in the example.
>
>
>
> I'm not sure how it could block an outstanding task since the closed
> channel is called before the Lock(), so no additional calls to RLock will
> be made at that point, and the Lock will just wait until all of the RLocks
> are complete.
>
>
>
> Regarding your testing strategy, I do like it better than any of my
> current strategy; however, There still is a chance that a task could
> complete between lines 90 and 91:
>
>
>
> h.Close()
>
> events <- ALL_TASKS_FINISHED
>
>
>
> So this doesn't solve the racy-ness I'm concerned about unless you put an
> arbitrary sleep in the handlers, which I'm trying to avoid.
>
>
> On Tuesday, 13 September 2016 12:34:17 UTC-7, aro...@gmail.com wrote:
>
> The WaitGroup is better than the lock approach, since the lock approach
> could block an outstanding task.  The key to using waitgroups is to call
> Add() outside of goroutines that might call done:
>
>
>
> https://play.golang.org/p/QVWoy8fCmI
>
> On Tuesday, September 13, 2016 at 12:19:16 PM UTC-7, Evan Digby 

Re: [go-nuts] Having difficulty testing this "cleanly"

[Evan Digby]

Hi Augusto,

I do owe you an apology. Our production code has the check on the closed 
channel after the RLock, not before. In this case you are 100% correct.

This was a bad transcription of the production code to the toy example.

Thanks again,

Evan

On Tuesday, 13 September 2016 13:24:03 UTC-7, aro...@gmail.com wrote:
>
> The mutex approach is fundamentally broken because you can't guarantee 
> that the tasks are all started (and have a read-lock acquired) before you 
> call close.
>
> Consider:
> h.Handle(...)  <-- gets past the closed channel check, calls go ..., 
> butthe goroutine doesn't execute yet.
> h.Close() <-- closes the close channel, Locks and Unlocks,returns.
> ...now the goroutine executes and acquires the read lock.
>
> So really, if you can't control the Handle() function, you need two 
> WaitGroups:  one to verify that all goroutines have started before shutting 
> down the task handler and a second one for all goroutines to have finished. 
>  However, it's tricky if we don't know the real use case.
>
> Sounds like you are trying to do graceful http shutdown.  Have you looked 
> at other libraries that do that?  If you don't have a way to account for 
> the time between Handle(..) is called and the goroutine starts, you always 
> might miss a task that got called near the time Close() was called.
>
> - Augusto
>
>
> On Tuesday, September 13, 2016 at 12:50:50 PM UTC-7, Evan Digby wrote:
>>
>> Hi Aroman,
>>
>> Your approach using the WaitGroup is definitely better in this toy 
>> example. The reason I didn't use the WaitGroup is because the non-toy 
>> example is wrapping the HTTP Server handler. I have no way to inject an 
>> "add" before the goroutine is created since that's handled by Go's HTTP 
>> Server without re-implementing the accept->handle loop using the listener. 
>>
>> Apologies for not giving the full context in the example.  
>>
>> I'm not sure how it could block an outstanding task since the closed 
>> channel is called before the Lock(), so no additional calls to RLock will 
>> be made at that point, and the Lock will just wait until all of the RLocks 
>> are complete.
>>
>> Regarding your testing strategy, I do like it better than any of my 
>> current strategy; however, There still is a chance that a task could 
>> complete between lines 90 and 91:
>>
>> h.Close()
>> events <- ALL_TASKS_FINISHED
>>
>> So this doesn't solve the racy-ness I'm concerned about unless you put an 
>> arbitrary sleep in the handlers, which I'm trying to avoid. 
>>
>> On Tuesday, 13 September 2016 12:34:17 UTC-7, aro...@gmail.com wrote:
>>>
>>> The WaitGroup is better than the lock approach, since the lock approach 
>>> could block an outstanding task.  The key to using waitgroups is to call 
>>> Add() outside of goroutines that might call done:
>>>
>>> https://play.golang.org/p/QVWoy8fCmI
>>>
>>> On Tuesday, September 13, 2016 at 12:19:16 PM UTC-7, Evan Digby wrote:

 Hi John,

 Thanks for the reply. I've tried many incarnations that include 
 WaitGroups; however, none seem to achieve the desired result. 

 If I add a WaitGroup with a defer done in the handler, and then wait 
 after the Close() then the test itself implements the requirement and 
 won't 
 protect from future refactors. There's no way to test that a WaitGroup is 
 done without waiting for it, and even if there was it would be racy 
 because 
 between the Close() and WaitGroup wait call tasks could complete. If I 
 wrapped the wait and the done in goroutines to see which one happened 
 first, also racy. 

 If you have something else in mind can you elaborate on how it would 
 help in this case?

 Thanks again!

 Evan

 On Tuesday, 13 September 2016 12:01:29 UTC-7, John Souvestre wrote:
>
> Have you considered using a sync.WaitGroup?
>
>  
>
> John
>
> John Souvestre - New Orleans LA
>
>  
>
> *From:* golan...@googlegroups.com [mailto:golan...@googlegroups.com] *On 
> Behalf Of *Evan Digby
> *Sent:* 2016 September 13, Tue 13:56
> *To:* golang-nuts
> *Subject:* [go-nuts] Having difficulty testing this "cleanly"
>
>  
>
> Has anyone come across a good way, non-racy way to ensure that N tasks 
> are guaranteed to be completed after a function is called? Essentially I 
> have a “Close” function that must be guaranteed to block until all tasks 
> are finished. Achieving this was pretty simple: wrap each task in an 
> RLock, 
> and then a Lock on close. 
>
>  
>
> Example: https://play.golang.org/p/7lhBPUhkUE
>
>  
>
> Now I want to write a solid test to guarantee Close will meet that 
> requirement of all tasks must finish first for posterity. In that 
> example, 
> try commenting out the RLock/RUnlock on lines 25/26. You'll see that it 
> no 
> longer outputs many, if any, lines. I'm 

Re: [go-nuts] Having difficulty testing this "cleanly"

[Evan Digby]

Hi John/Egon/Augusto,

I should point out is that all we need to guarantee (barring abnormal 
termination of course) is that once a task starts processing, it finishes. 
Partially processed messages are bad, but http requests that don't result 
in a message being processed at all are okay.

We don't need to guarantee that the result of every Accept in the HTTP 
server results in a processed message. We handle this on the OPS side by 
ensuring we stop sending requests to that instance before terminating the 
process. We just want to make sure, at that point, that the messages which 
did make it to the handler are flushed.

So the case where:

h.Handle(...)  <-- gets past the closed channel check, calls go ..., butthe 
goroutine doesn't execute yet.
h.Close() <-- closes the close channel, Locks and Unlocks,returns.
...now the goroutine executes and acquires the read lock.

We actually don't care if "Handle" completes in this example. We only care 
if that our task handler starts processing a message that it completes the 
processing.

Thanks again,

Evan

On Tuesday, 13 September 2016 13:24:03 UTC-7, aro...@gmail.com wrote:
>
> The mutex approach is fundamentally broken because you can't guarantee 
> that the tasks are all started (and have a read-lock acquired) before you 
> call close.
>
> Consider:
> h.Handle(...)  <-- gets past the closed channel check, calls go ..., 
> butthe goroutine doesn't execute yet.
> h.Close() <-- closes the close channel, Locks and Unlocks,returns.
> ...now the goroutine executes and acquires the read lock.
>
> So really, if you can't control the Handle() function, you need two 
> WaitGroups:  one to verify that all goroutines have started before shutting 
> down the task handler and a second one for all goroutines to have finished. 
>  However, it's tricky if we don't know the real use case.
>
> Sounds like you are trying to do graceful http shutdown.  Have you looked 
> at other libraries that do that?  If you don't have a way to account for 
> the time between Handle(..) is called and the goroutine starts, you always 
> might miss a task that got called near the time Close() was called.
>
> - Augusto
>
>
> On Tuesday, September 13, 2016 at 12:50:50 PM UTC-7, Evan Digby wrote:
>>
>> Hi Aroman,
>>
>> Your approach using the WaitGroup is definitely better in this toy 
>> example. The reason I didn't use the WaitGroup is because the non-toy 
>> example is wrapping the HTTP Server handler. I have no way to inject an 
>> "add" before the goroutine is created since that's handled by Go's HTTP 
>> Server without re-implementing the accept->handle loop using the listener. 
>>
>> Apologies for not giving the full context in the example.  
>>
>> I'm not sure how it could block an outstanding task since the closed 
>> channel is called before the Lock(), so no additional calls to RLock will 
>> be made at that point, and the Lock will just wait until all of the RLocks 
>> are complete.
>>
>> Regarding your testing strategy, I do like it better than any of my 
>> current strategy; however, There still is a chance that a task could 
>> complete between lines 90 and 91:
>>
>> h.Close()
>> events <- ALL_TASKS_FINISHED
>>
>> So this doesn't solve the racy-ness I'm concerned about unless you put an 
>> arbitrary sleep in the handlers, which I'm trying to avoid. 
>>
>> On Tuesday, 13 September 2016 12:34:17 UTC-7, aro...@gmail.com wrote:
>>>
>>> The WaitGroup is better than the lock approach, since the lock approach 
>>> could block an outstanding task.  The key to using waitgroups is to call 
>>> Add() outside of goroutines that might call done:
>>>
>>> https://play.golang.org/p/QVWoy8fCmI
>>>
>>> On Tuesday, September 13, 2016 at 12:19:16 PM UTC-7, Evan Digby wrote:

 Hi John,

 Thanks for the reply. I've tried many incarnations that include 
 WaitGroups; however, none seem to achieve the desired result. 

 If I add a WaitGroup with a defer done in the handler, and then wait 
 after the Close() then the test itself implements the requirement and 
 won't 
 protect from future refactors. There's no way to test that a WaitGroup is 
 done without waiting for it, and even if there was it would be racy 
 because 
 between the Close() and WaitGroup wait call tasks could complete. If I 
 wrapped the wait and the done in goroutines to see which one happened 
 first, also racy. 

 If you have something else in mind can you elaborate on how it would 
 help in this case?

 Thanks again!

 Evan

 On Tuesday, 13 September 2016 12:01:29 UTC-7, John Souvestre wrote:
>
> Have you considered using a sync.WaitGroup?
>
>  
>
> John
>
> John Souvestre - New Orleans LA
>
>  
>
> *From:* golan...@googlegroups.com [mailto:golan...@googlegroups.com] *On 
> Behalf Of *Evan Digby
> *Sent:* 2016 September 13, Tue 13:56
> *To:* golang-nuts
> *Subject:* 

[go-nuts] Re: Having difficulty testing this "cleanly"

[Evan Digby]

Hi Egon,

I appreciate the effort you've (and everyone has) put in thus-far!

Correct me if I'm wrong, but I think you're pointing out the race condition 
John pointed. That's a tradeoff we've accepted so we don't need to 
reimplement much of http.Serve so that we can add a waitgroup Add or Rlock 
before the goroutine is spawned (among other sync requirements to ensure no 
new connections are accepted, etc).

Thanks again,

Evan



On Tuesday, 13 September 2016 13:11:17 UTC-7, Egon wrote:
>
> On Tuesday, 13 September 2016 22:52:27 UTC+3, Evan Digby wrote:
>>
>> Hi Egon,
>>
>> This is essentially the strategy I'm taking; however, I am hoping to 
>> avoid the "Sleep and Pray" method. Reliable in practice, but not 
>> guaranteed. Also in a CI of thousands of tests, adding time arbitrarily can 
>> extend out the time it takes to test quite a bit.
>>
>> That said, if a sleep is the only way, a sleep is the only way. I hope it 
>> isn't!
>>
>
> You would need to modify the handler to make it work other ways.
> Alternatively you need to write a code-rewriter for injecting code into 
> the handler.
>
> The only reliable concurrent software I've seen is one that is easy to 
> understand, that is backed by a proof (either formal or informal).
>
> I took a deeper look into the taskHandler code -- and it doesn't finish 
> all the tasks before everything:
>
> I.e. 
>
> R1: Line 24 // go func is delayed for some reason
> R2: Line 35 // close is called in main
> R2: Line 36
> R2: Line 37
> R1: Line 25 // go func is started
> R1: Line 28 // task is executed
> R1: Line 26
>
> ATM. too tired to implement a proper solution, will take a look at it 
> tomorrow.
>
> + Egon
>
>
>> Thanks!
>>
>> Evan
>>
>> On Tuesday, 13 September 2016 12:47:40 UTC-7, Egon wrote:
>>>
>>> counter := intr64(N)
>>> release := make(chan struct{})
>>>
>>> ...
>>> for i := 0; i < N; i ++ {
>>>   h.Handle(func() {
>>>   <-release
>>>   atomic.AddInt64(, -1)
>>>   })
>>> }
>>> ...
>>>
>>> go func(){
>>> time.Sleep(time.Millisecond) // so we would certainly hit h.Close, 
>>> before we continue
>>> for i := 0; i < N; i++ { release <- struct{}{}; 
>>> time.Sleep(time.Millisecond) }
>>> // alternatively use runtime.Gosched() instead of Sleep
>>> }()
>>>
>>> h.Close()
>>>
>>> if atomic.LoadInt64() > 0 {
>>> // fail
>>> }
>>>
>>> It's not completely fool-proof, but should work well enough in practice.
>>>
>>> On Tuesday, 13 September 2016 21:56:08 UTC+3, Evan Digby wrote:

 Has anyone come across a good way, non-racy way to ensure that N tasks 
 are guaranteed to be completed after a function is called? Essentially I 
 have a “Close” function that must be guaranteed to block until all tasks 
 are finished. Achieving this was pretty simple: wrap each task in an 
 RLock, 
 and then a Lock on close. 

 Example: https://play.golang.org/p/7lhBPUhkUE

 Now I want to write a solid test to guarantee Close will meet that 
 requirement of all tasks must finish first for posterity. In that example, 
 try commenting out the RLock/RUnlock on lines 25/26. You'll see that it no 
 longer outputs many, if any, lines. I'm trying to prevent that from 
 happening in the future by some cowboy refactor!

>>>
 All of the ways I can come up with involve Sleeping or launching more 
 tasks than I _think_ can be finished in time--obviously not good!

 I feel like I must be missing some obvious way to test this and I'll 
 end up feeling silly once someone replies with the solution. I'm okay with 
 that!

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [go-nuts] Having difficulty testing this "cleanly"

[John Souvestre]

Ø  The RW mutex is used by the implementation to guarantee that all handlers 
are complete before we return from "Close", which is what I'm attempting to 
test.

 

I’m not sure that the mutex is going to work like you want.  There’s no 
guarantee that the h.RLock() in the goroutine will execute in a timely fashion. 
 I think that you’d have to call it before the goroutine.

 

John

John Souvestre - New Orleans LA

 

From: golang-nuts@googlegroups.com [mailto:golang-nuts@googlegroups.com] On 
Behalf Of Evan Digby
Sent: 2016 September 13, Tue 14:59
To: golang-nuts
Subject: Re: [go-nuts] Having difficulty testing this "cleanly"

 

Hi John,

 

Consider the "main" function a test, and not a sane usage of the handler.

 

The multiple sync methods in "main" are trying to accomplish:

 

1) Ensure all tasks have started (allStarted waitgroup)

2) Wait until we're ready to call "Done" before we proceed with the task 
(timeToFinish channel block)

 

These synchronizations wouldn't exist in actual usage. They're just test setup.

 

The RW mutex is used by the implementation to guarantee that all handlers are 
complete before we return from "Close", which is what I'm attempting to test.

 

Thanks again!

 

 

 


On Tuesday, 13 September 2016 12:49:07 UTC-7, John Souvestre wrote:

Hi Evan.

 

I looked at the code you posted.  I’m having a problem seeing exactly what you 
are trying to do.  Also, it appears to be using multiple synchronization 
techniques: Mutex, WaitGroup, and 2 blocking channels.

 

Am I correct in guessing that you want to know how long it took to get all of 
the tasks running?  Then you want to wait till all of the tasks are complete?  
If so, I’m thinking that using 2 WaitGroups would do the job – one to show when 
all the tasks are running and one to show when all the tasks are done.  No 
mutex and no blocking channels.

 

John

John Souvestre - New Orleans LA

 

From: golan...@googlegroups.com   
[mailto:golan...@googlegroups.com  ] On Behalf Of Evan Digby
Sent: 2016 September 13, Tue 14:19
To: golang-nuts
Subject: Re: [go-nuts] Having difficulty testing this "cleanly"

 

Hi John,

 

Thanks for the reply. I've tried many incarnations that include WaitGroups; 
however, none seem to achieve the desired result. 

 

If I add a WaitGroup with a defer done in the handler, and then wait after the 
Close() then the test itself implements the requirement and won't protect from 
future refactors. There's no way to test that a WaitGroup is done without 
waiting for it, and even if there was it would be racy because between the 
Close() and WaitGroup wait call tasks could complete. If I wrapped the wait and 
the done in goroutines to see which one happened first, also racy. 

 

If you have something else in mind can you elaborate on how it would help in 
this case?

 

Thanks again!

 

Evan


On Tuesday, 13 September 2016 12:01:29 UTC-7, John Souvestre wrote:

Have you considered using a sync.WaitGroup?

 

John

John Souvestre - New Orleans LA

 

From: golan...@googlegroups.com [mailto:golan...@googlegroups.com] On Behalf Of 
Evan Digby
Sent: 2016 September 13, Tue 13:56
To: golang-nuts
Subject: [go-nuts] Having difficulty testing this "cleanly"

 

Has anyone come across a good way, non-racy way to ensure that N tasks are 
guaranteed to be completed after a function is called? Essentially I have a 
“Close” function that must be guaranteed to block until all tasks are finished. 
Achieving this was pretty simple: wrap each task in an RLock, and then a Lock 
on close. 

 

Example: https://play.golang.org/p/7lhBPUhkUE

 

Now I want to write a solid test to guarantee Close will meet that requirement 
of all tasks must finish first for posterity. In that example, try commenting 
out the RLock/RUnlock on lines 25/26. You'll see that it no longer outputs 
many, if any, lines. I'm trying to prevent that from happening in the future by 
some cowboy refactor!

 

All of the ways I can come up with involve Sleeping or launching more tasks 
than I _think_ can be finished in time--obviously not good!

 

I feel like I must be missing some obvious way to test this and I'll end up 
feeling silly once someone replies with the solution. I'm okay with that!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts...@googlegroups.com  .
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: [go-nuts] Having difficulty testing this "cleanly"

[Evan Digby]

Hi John,

Consider the "main" function a test, and not a sane usage of the handler.

The multiple sync methods in "main" are trying to accomplish:

1) Ensure all tasks have started (allStarted waitgroup)
2) Wait until we're ready to call "Done" before we proceed with the task 
(timeToFinish channel block)

These synchronizations wouldn't exist in actual usage. They're just test 
setup.

The RW mutex is used by the implementation to guarantee that all handlers 
are complete before we return from "Close", which is what I'm attempting to 
test.

Thanks again!




On Tuesday, 13 September 2016 12:49:07 UTC-7, John Souvestre wrote:
>
> Hi Evan.
>
>  
>
> I looked at the code you posted.  I’m having a problem seeing exactly what 
> you are trying to do.  Also, it appears to be using multiple 
> synchronization techniques: Mutex, WaitGroup, and 2 blocking channels.
>
>  
>
> Am I correct in guessing that you want to know how long it took to get all 
> of the tasks running?  Then you want to wait till all of the tasks are 
> complete?  If so, I’m thinking that using 2 WaitGroups would do the job – 
> one to show when all the tasks are running and one to show when all the 
> tasks are done.  No mutex and no blocking channels.
>
>  
>
> John
>
> John Souvestre - New Orleans LA
>
>  
>
> *From:* golan...@googlegroups.com  [mailto:
> golan...@googlegroups.com ] *On Behalf Of *Evan Digby
> *Sent:* 2016 September 13, Tue 14:19
> *To:* golang-nuts
> *Subject:* Re: [go-nuts] Having difficulty testing this "cleanly"
>
>  
>
> Hi John,
>
>  
>
> Thanks for the reply. I've tried many incarnations that include 
> WaitGroups; however, none seem to achieve the desired result. 
>
>  
>
> If I add a WaitGroup with a defer done in the handler, and then wait after 
> the Close() then the test itself implements the requirement and won't 
> protect from future refactors. There's no way to test that a WaitGroup is 
> done without waiting for it, and even if there was it would be racy because 
> between the Close() and WaitGroup wait call tasks could complete. If I 
> wrapped the wait and the done in goroutines to see which one happened 
> first, also racy. 
>
>  
>
> If you have something else in mind can you elaborate on how it would help 
> in this case?
>
>  
>
> Thanks again!
>
>  
>
> Evan
>
>
> On Tuesday, 13 September 2016 12:01:29 UTC-7, John Souvestre wrote:
>
> Have you considered using a sync.WaitGroup?
>
>  
>
> John
>
> John Souvestre - New Orleans LA
>
>  
>
> *From:* golan...@googlegroups.com [mailto:golan...@googlegroups.com] *On 
> Behalf Of *Evan Digby
> *Sent:* 2016 September 13, Tue 13:56
> *To:* golang-nuts
> *Subject:* [go-nuts] Having difficulty testing this "cleanly"
>
>  
>
> Has anyone come across a good way, non-racy way to ensure that N tasks are 
> guaranteed to be completed after a function is called? Essentially I have a 
> “Close” function that must be guaranteed to block until all tasks are 
> finished. Achieving this was pretty simple: wrap each task in an RLock, and 
> then a Lock on close. 
>
>  
>
> Example: https://play.golang.org/p/7lhBPUhkUE
>
>  
>
> Now I want to write a solid test to guarantee Close will meet that 
> requirement of all tasks must finish first for posterity. In that example, 
> try commenting out the RLock/RUnlock on lines 25/26. You'll see that it no 
> longer outputs many, if any, lines. I'm trying to prevent that from 
> happening in the future by some cowboy refactor!
>
>  
>
> All of the ways I can come up with involve Sleeping or launching more 
> tasks than I _think_ can be finished in time--obviously not good!
>
>  
>
> I feel like I must be missing some obvious way to test this and I'll end 
> up feeling silly once someone replies with the solution. I'm okay with that!
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [go-nuts] Having difficulty testing this "cleanly"

[John Souvestre]

Hi again.

 

Ø  There's no way to test that a WaitGroup is done without waiting for it, and 
even if there was it would be racy because between the Close() and WaitGroup 
wait call tasks could complete.

 

If you don’t mind being blocked, then Wait is just what you want.  Since it’s 
occurring in the same function (main) that the Add is taking place earlier, 
there is no race.  

 

John

John Souvestre - New Orleans LA

 

From: golang-nuts@googlegroups.com [mailto:golang-nuts@googlegroups.com] On 
Behalf Of Evan Digby
Sent: 2016 September 13, Tue 14:19
To: golang-nuts
Subject: Re: [go-nuts] Having difficulty testing this "cleanly"

 

Hi John,

 

Thanks for the reply. I've tried many incarnations that include WaitGroups; 
however, none seem to achieve the desired result. 

 

If I add a WaitGroup with a defer done in the handler, and then wait after the 
Close() then the test itself implements the requirement and won't protect from 
future refactors. There's no way to test that a WaitGroup is done without 
waiting for it, and even if there was it would be racy because between the 
Close() and WaitGroup wait call tasks could complete. If I wrapped the wait and 
the done in goroutines to see which one happened first, also racy. 

 

If you have something else in mind can you elaborate on how it would help in 
this case?

 

Thanks again!

 

Evan


On Tuesday, 13 September 2016 12:01:29 UTC-7, John Souvestre wrote:

Have you considered using a sync.WaitGroup?

 

John

John Souvestre - New Orleans LA

 

From: golan...@googlegroups.com   
[mailto:golan...@googlegroups.com  ] On Behalf Of Evan Digby
Sent: 2016 September 13, Tue 13:56
To: golang-nuts
Subject: [go-nuts] Having difficulty testing this "cleanly"

 

Has anyone come across a good way, non-racy way to ensure that N tasks are 
guaranteed to be completed after a function is called? Essentially I have a 
“Close” function that must be guaranteed to block until all tasks are finished. 
Achieving this was pretty simple: wrap each task in an RLock, and then a Lock 
on close. 

 

Example: https://play.golang.org/p/7lhBPUhkUE

 

Now I want to write a solid test to guarantee Close will meet that requirement 
of all tasks must finish first for posterity. In that example, try commenting 
out the RLock/RUnlock on lines 25/26. You'll see that it no longer outputs 
many, if any, lines. I'm trying to prevent that from happening in the future by 
some cowboy refactor!

 

All of the ways I can come up with involve Sleeping or launching more tasks 
than I _think_ can be finished in time--obviously not good!

 

I feel like I must be missing some obvious way to test this and I'll end up 
feeling silly once someone replies with the solution. I'm okay with that!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts...@googlegroups.com  .
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Having difficulty testing this "cleanly"

[Evan Digby]

Hi Egon,

This is essentially the strategy I'm taking; however, I am hoping to avoid 
the "Sleep and Pray" method. Reliable in practice, but not guaranteed. Also 
in a CI of thousands of tests, adding time arbitrarily can extend out the 
time it takes to test quite a bit.

That said, if a sleep is the only way, a sleep is the only way. I hope it 
isn't!

Thanks!

Evan

On Tuesday, 13 September 2016 12:47:40 UTC-7, Egon wrote:
>
> counter := intr64(N)
> release := make(chan struct{})
>
> ...
> for i := 0; i < N; i ++ {
>   h.Handle(func() {
>   <-release
>   atomic.AddInt64(, -1)
>   })
> }
> ...
>
> go func(){
> time.Sleep(time.Millisecond) // so we would certainly hit h.Close, 
> before we continue
> for i := 0; i < N; i++ { release <- struct{}{}; 
> time.Sleep(time.Millisecond) }
> // alternatively use runtime.Gosched() instead of Sleep
> }()
>
> h.Close()
>
> if atomic.LoadInt64() > 0 {
> // fail
> }
>
> It's not completely fool-proof, but should work well enough in practice.
>
> On Tuesday, 13 September 2016 21:56:08 UTC+3, Evan Digby wrote:
>>
>> Has anyone come across a good way, non-racy way to ensure that N tasks 
>> are guaranteed to be completed after a function is called? Essentially I 
>> have a “Close” function that must be guaranteed to block until all tasks 
>> are finished. Achieving this was pretty simple: wrap each task in an RLock, 
>> and then a Lock on close. 
>>
>> Example: https://play.golang.org/p/7lhBPUhkUE
>>
>> Now I want to write a solid test to guarantee Close will meet that 
>> requirement of all tasks must finish first for posterity. In that example, 
>> try commenting out the RLock/RUnlock on lines 25/26. You'll see that it no 
>> longer outputs many, if any, lines. I'm trying to prevent that from 
>> happening in the future by some cowboy refactor!
>>
>
>> All of the ways I can come up with involve Sleeping or launching more 
>> tasks than I _think_ can be finished in time--obviously not good!
>>
>> I feel like I must be missing some obvious way to test this and I'll end 
>> up feeling silly once someone replies with the solution. I'm okay with that!
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Having difficulty testing this "cleanly"

[Evan Digby]

Hi Aroman,

Your approach using the WaitGroup is definitely better in this toy example. 
The reason I didn't use the WaitGroup is because the non-toy example is 
wrapping the HTTP Server handler. I have no way to inject an "add" before 
the goroutine is created since that's handled by Go's HTTP Server without 
re-implementing the accept->handle loop using the listener. 

Apologies for not giving the full context in the example.  

I'm not sure how it could block an outstanding task since the closed 
channel is called before the Lock(), so no additional calls to RLock will 
be made at that point, and the Lock will just wait until all of the RLocks 
are complete.

Regarding your testing strategy, I do like it better than any of my current 
strategy; however, There still is a chance that a task could complete 
between lines 90 and 91:

h.Close()
events <- ALL_TASKS_FINISHED

So this doesn't solve the racy-ness I'm concerned about unless you put an 
arbitrary sleep in the handlers, which I'm trying to avoid. 

On Tuesday, 13 September 2016 12:34:17 UTC-7, aro...@gmail.com wrote:
>
> The WaitGroup is better than the lock approach, since the lock approach 
> could block an outstanding task.  The key to using waitgroups is to call 
> Add() outside of goroutines that might call done:
>
> https://play.golang.org/p/QVWoy8fCmI
>
> On Tuesday, September 13, 2016 at 12:19:16 PM UTC-7, Evan Digby wrote:
>>
>> Hi John,
>>
>> Thanks for the reply. I've tried many incarnations that include 
>> WaitGroups; however, none seem to achieve the desired result. 
>>
>> If I add a WaitGroup with a defer done in the handler, and then wait 
>> after the Close() then the test itself implements the requirement and won't 
>> protect from future refactors. There's no way to test that a WaitGroup is 
>> done without waiting for it, and even if there was it would be racy because 
>> between the Close() and WaitGroup wait call tasks could complete. If I 
>> wrapped the wait and the done in goroutines to see which one happened 
>> first, also racy. 
>>
>> If you have something else in mind can you elaborate on how it would help 
>> in this case?
>>
>> Thanks again!
>>
>> Evan
>>
>> On Tuesday, 13 September 2016 12:01:29 UTC-7, John Souvestre wrote:
>>>
>>> Have you considered using a sync.WaitGroup?
>>>
>>>  
>>>
>>> John
>>>
>>> John Souvestre - New Orleans LA
>>>
>>>  
>>>
>>> *From:* golan...@googlegroups.com [mailto:golan...@googlegroups.com] *On 
>>> Behalf Of *Evan Digby
>>> *Sent:* 2016 September 13, Tue 13:56
>>> *To:* golang-nuts
>>> *Subject:* [go-nuts] Having difficulty testing this "cleanly"
>>>
>>>  
>>>
>>> Has anyone come across a good way, non-racy way to ensure that N tasks 
>>> are guaranteed to be completed after a function is called? Essentially I 
>>> have a “Close” function that must be guaranteed to block until all tasks 
>>> are finished. Achieving this was pretty simple: wrap each task in an RLock, 
>>> and then a Lock on close. 
>>>
>>>  
>>>
>>> Example: https://play.golang.org/p/7lhBPUhkUE
>>>
>>>  
>>>
>>> Now I want to write a solid test to guarantee Close will meet that 
>>> requirement of all tasks must finish first for posterity. In that example, 
>>> try commenting out the RLock/RUnlock on lines 25/26. You'll see that it no 
>>> longer outputs many, if any, lines. I'm trying to prevent that from 
>>> happening in the future by some cowboy refactor!
>>>
>>>  
>>>
>>> All of the ways I can come up with involve Sleeping or launching more 
>>> tasks than I _think_ can be finished in time--obviously not good!
>>>
>>>  
>>>
>>> I feel like I must be missing some obvious way to test this and I'll end 
>>> up feeling silly once someone replies with the solution. I'm okay with that!
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to golang-nuts...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [go-nuts] Having difficulty testing this "cleanly"

[John Souvestre]

Hi Evan.

 

I looked at the code you posted.  I’m having a problem seeing exactly what you 
are trying to do.  Also, it appears to be using multiple synchronization 
techniques: Mutex, WaitGroup, and 2 blocking channels.

 

Am I correct in guessing that you want to know how long it took to get all of 
the tasks running?  Then you want to wait till all of the tasks are complete?  
If so, I’m thinking that using 2 WaitGroups would do the job – one to show when 
all the tasks are running and one to show when all the tasks are done.  No 
mutex and no blocking channels.

 

John

John Souvestre - New Orleans LA

 

From: golang-nuts@googlegroups.com [mailto:golang-nuts@googlegroups.com] On 
Behalf Of Evan Digby
Sent: 2016 September 13, Tue 14:19
To: golang-nuts
Subject: Re: [go-nuts] Having difficulty testing this "cleanly"

 

Hi John,

 

Thanks for the reply. I've tried many incarnations that include WaitGroups; 
however, none seem to achieve the desired result. 

 

If I add a WaitGroup with a defer done in the handler, and then wait after the 
Close() then the test itself implements the requirement and won't protect from 
future refactors. There's no way to test that a WaitGroup is done without 
waiting for it, and even if there was it would be racy because between the 
Close() and WaitGroup wait call tasks could complete. If I wrapped the wait and 
the done in goroutines to see which one happened first, also racy. 

 

If you have something else in mind can you elaborate on how it would help in 
this case?

 

Thanks again!

 

Evan


On Tuesday, 13 September 2016 12:01:29 UTC-7, John Souvestre wrote:

Have you considered using a sync.WaitGroup?

 

John

John Souvestre - New Orleans LA

 

From: golan...@googlegroups.com   
[mailto:golan...@googlegroups.com  ] On Behalf Of Evan Digby
Sent: 2016 September 13, Tue 13:56
To: golang-nuts
Subject: [go-nuts] Having difficulty testing this "cleanly"

 

Has anyone come across a good way, non-racy way to ensure that N tasks are 
guaranteed to be completed after a function is called? Essentially I have a 
“Close” function that must be guaranteed to block until all tasks are finished. 
Achieving this was pretty simple: wrap each task in an RLock, and then a Lock 
on close. 

 

Example: https://play.golang.org/p/7lhBPUhkUE

 

Now I want to write a solid test to guarantee Close will meet that requirement 
of all tasks must finish first for posterity. In that example, try commenting 
out the RLock/RUnlock on lines 25/26. You'll see that it no longer outputs 
many, if any, lines. I'm trying to prevent that from happening in the future by 
some cowboy refactor!

 

All of the ways I can come up with involve Sleeping or launching more tasks 
than I _think_ can be finished in time--obviously not good!

 

I feel like I must be missing some obvious way to test this and I'll end up 
feeling silly once someone replies with the solution. I'm okay with that!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts...@googlegroups.com  .
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Having difficulty testing this "cleanly"

[Egon]

counter := intr64(N)
release := make(chan struct{})

...
for i := 0; i < N; i ++ {
  h.Handle(func() {
  <-release
  atomic.AddInt64(, -1)
  })
}
...

go func(){
time.Sleep(time.Millisecond) // so we would certainly hit h.Close, 
before we continue
for i := 0; i < N; i++ { release <- struct{}{}; 
time.Sleep(time.Millisecond) }
// alternatively use runtime.Gosched() instead of Sleep
}()

h.Close()

if atomic.LoadInt64() > 0 {
// fail
}

It's not completely fool-proof, but should work well enough in practice.

On Tuesday, 13 September 2016 21:56:08 UTC+3, Evan Digby wrote:
>
> Has anyone come across a good way, non-racy way to ensure that N tasks are 
> guaranteed to be completed after a function is called? Essentially I have a 
> “Close” function that must be guaranteed to block until all tasks are 
> finished. Achieving this was pretty simple: wrap each task in an RLock, and 
> then a Lock on close. 
>
> Example: https://play.golang.org/p/7lhBPUhkUE
>
> Now I want to write a solid test to guarantee Close will meet that 
> requirement of all tasks must finish first for posterity. In that example, 
> try commenting out the RLock/RUnlock on lines 25/26. You'll see that it no 
> longer outputs many, if any, lines. I'm trying to prevent that from 
> happening in the future by some cowboy refactor!
>

> All of the ways I can come up with involve Sleeping or launching more 
> tasks than I _think_ can be finished in time--obviously not good!
>
> I feel like I must be missing some obvious way to test this and I'll end 
> up feeling silly once someone replies with the solution. I'm okay with that!
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Having difficulty testing this "cleanly"

[aroman]

The WaitGroup is better than the lock approach, since the lock approach 
could block an outstanding task.  The key to using waitgroups is to call 
Add() outside of goroutines that might call done:

https://play.golang.org/p/QVWoy8fCmI

On Tuesday, September 13, 2016 at 12:19:16 PM UTC-7, Evan Digby wrote:
>
> Hi John,
>
> Thanks for the reply. I've tried many incarnations that include 
> WaitGroups; however, none seem to achieve the desired result. 
>
> If I add a WaitGroup with a defer done in the handler, and then wait after 
> the Close() then the test itself implements the requirement and won't 
> protect from future refactors. There's no way to test that a WaitGroup is 
> done without waiting for it, and even if there was it would be racy because 
> between the Close() and WaitGroup wait call tasks could complete. If I 
> wrapped the wait and the done in goroutines to see which one happened 
> first, also racy. 
>
> If you have something else in mind can you elaborate on how it would help 
> in this case?
>
> Thanks again!
>
> Evan
>
> On Tuesday, 13 September 2016 12:01:29 UTC-7, John Souvestre wrote:
>>
>> Have you considered using a sync.WaitGroup?
>>
>>  
>>
>> John
>>
>> John Souvestre - New Orleans LA
>>
>>  
>>
>> *From:* golan...@googlegroups.com [mailto:golan...@googlegroups.com] *On 
>> Behalf Of *Evan Digby
>> *Sent:* 2016 September 13, Tue 13:56
>> *To:* golang-nuts
>> *Subject:* [go-nuts] Having difficulty testing this "cleanly"
>>
>>  
>>
>> Has anyone come across a good way, non-racy way to ensure that N tasks 
>> are guaranteed to be completed after a function is called? Essentially I 
>> have a “Close” function that must be guaranteed to block until all tasks 
>> are finished. Achieving this was pretty simple: wrap each task in an RLock, 
>> and then a Lock on close. 
>>
>>  
>>
>> Example: https://play.golang.org/p/7lhBPUhkUE
>>
>>  
>>
>> Now I want to write a solid test to guarantee Close will meet that 
>> requirement of all tasks must finish first for posterity. In that example, 
>> try commenting out the RLock/RUnlock on lines 25/26. You'll see that it no 
>> longer outputs many, if any, lines. I'm trying to prevent that from 
>> happening in the future by some cowboy refactor!
>>
>>  
>>
>> All of the ways I can come up with involve Sleeping or launching more 
>> tasks than I _think_ can be finished in time--obviously not good!
>>
>>  
>>
>> I feel like I must be missing some obvious way to test this and I'll end 
>> up feeling silly once someone replies with the solution. I'm okay with that!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Having difficulty testing this "cleanly"

[Evan Digby]

Hi John,

Thanks for the reply. I've tried many incarnations that include WaitGroups; 
however, none seem to achieve the desired result. 

If I add a WaitGroup with a defer done in the handler, and then wait after 
the Close() then the test itself implements the requirement and won't 
protect from future refactors. There's no way to test that a WaitGroup is 
done without waiting for it, and even if there was it would be racy because 
between the Close() and WaitGroup wait call tasks could complete. If I 
wrapped the wait and the done in goroutines to see which one happened 
first, also racy. 

If you have something else in mind can you elaborate on how it would help 
in this case?

Thanks again!

Evan

On Tuesday, 13 September 2016 12:01:29 UTC-7, John Souvestre wrote:
>
> Have you considered using a sync.WaitGroup?
>
>  
>
> John
>
> John Souvestre - New Orleans LA
>
>  
>
> *From:* golan...@googlegroups.com  [mailto:
> golan...@googlegroups.com ] *On Behalf Of *Evan Digby
> *Sent:* 2016 September 13, Tue 13:56
> *To:* golang-nuts
> *Subject:* [go-nuts] Having difficulty testing this "cleanly"
>
>  
>
> Has anyone come across a good way, non-racy way to ensure that N tasks are 
> guaranteed to be completed after a function is called? Essentially I have a 
> “Close” function that must be guaranteed to block until all tasks are 
> finished. Achieving this was pretty simple: wrap each task in an RLock, and 
> then a Lock on close. 
>
>  
>
> Example: https://play.golang.org/p/7lhBPUhkUE
>
>  
>
> Now I want to write a solid test to guarantee Close will meet that 
> requirement of all tasks must finish first for posterity. In that example, 
> try commenting out the RLock/RUnlock on lines 25/26. You'll see that it no 
> longer outputs many, if any, lines. I'm trying to prevent that from 
> happening in the future by some cowboy refactor!
>
>  
>
> All of the ways I can come up with involve Sleeping or launching more 
> tasks than I _think_ can be finished in time--obviously not good!
>
>  
>
> I feel like I must be missing some obvious way to test this and I'll end 
> up feeling silly once someone replies with the solution. I'm okay with that!
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Type Scope?

[aroman]

That's correct.  Method receivers are equivalent to just passing as an 
extra argument:

func Set(f fetcher, tname string) {
f.names = tname
}

In this case, it's obvious that Set will not affect the fetcher passed in 
at the call site, in contrast to

func Set(f *fetcher, tname string) {
f.names = tname
}

...which would have the desired effect.  Same with pointer receivers:

func (f fetcher) Set(tname string) {
f.names = tname
}
vs
func (f *fetcher) Set(tname string) {
f.names = tname
}

- Augusto

On Tuesday, September 13, 2016 at 12:14:29 PM UTC-7, Frank Davidson wrote:
>
> Thanks, guys!
>
> So, just for me to be clear, the line me.Set("It works!") in my original 
> code created a copy of the "me" fetcher, set its names field, and then 
> threw it away, leaving the original "me.name" with an unaltered, empty 
> string value?
>
>
> On Tuesday, September 13, 2016 at 2:41:46 PM UTC-4, Frank Davidson wrote:
>>
>> I think I'm having some type of mental lapse, but can anyone tell me why 
>> this doesn't print out "It works!" nor give me an error? It simply prints 
>> an empty string...
>>
>> https://play.golang.org/p/9oG0S8kfM3
>>
>> package main
>>
>> import "fmt"
>>
>> type fetcher struct { names string }
>>
>> func (f fetcher) Set(tname string) {
>> f.names = tname
>> }
>>
>> func main() {
>> var me fetcher
>> me.Set("It works!")
>> fmt.Println(me.names)
>> }
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Multiple-reader single-writer map access - is this lockless workaround safe?

[Ian Lance Taylor]

On Tue, Sep 13, 2016 at 9:36 AM, sqweek E.  wrote:
>
> The remaining examples sound indistinguishable from compiler bugs.

Developers of C/C++ compilers rigorously rely on the language
standard's definition of undefined behavior when optimizing programs.
This can lead to many behaviors that, when first encountered, are
often denounced as compiler bugs.  But the compiler's only promise is
to implement the language standard.

For example, until C++11 clarified that speculative stores were
forbidden, C/C++ compilers really did generate them, and it really did
baffle and confuse people and break programs including the Linux
kernel, but it was not wrong according to the then-relevant language
standards.  There are many similar cases.  It's easier to say that the
compiler is broken than it is to point to the rules that explain why
that is the case.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [go-nuts] Having difficulty testing this "cleanly"

[John Souvestre]

Have you considered using a sync.WaitGroup?

 

John

John Souvestre - New Orleans LA

 

From: golang-nuts@googlegroups.com [mailto:golang-nuts@googlegroups.com] On 
Behalf Of Evan Digby
Sent: 2016 September 13, Tue 13:56
To: golang-nuts
Subject: [go-nuts] Having difficulty testing this "cleanly"

 

Has anyone come across a good way, non-racy way to ensure that N tasks are 
guaranteed to be completed after a function is called? Essentially I have a 
“Close” function that must be guaranteed to block until all tasks are finished. 
Achieving this was pretty simple: wrap each task in an RLock, and then a Lock 
on close. 

 

Example: https://play.golang.org/p/7lhBPUhkUE

 

Now I want to write a solid test to guarantee Close will meet that requirement 
of all tasks must finish first for posterity. In that example, try commenting 
out the RLock/RUnlock on lines 25/26. You'll see that it no longer outputs 
many, if any, lines. I'm trying to prevent that from happening in the future by 
some cowboy refactor!

 

All of the ways I can come up with involve Sleeping or launching more tasks 
than I _think_ can be finished in time--obviously not good!

 

I feel like I must be missing some obvious way to test this and I'll end up 
feeling silly once someone replies with the solution. I'm okay with that!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Multiple-reader single-writer map access - is this lockless workaround safe?

[Henrik Johansson]

I think this will be an interesting read especially if you come from Java.

https://shipilev.net/blog/2016/close-encounters-of-jmm-kind/

On Tue, Sep 13, 2016, 19:08 sqweek E.  wrote:

> On Tuesday, September 13, 2016 at 10:57:47 PM UTC+8, Alan Donovan wrote:
>
>> On 13 September 2016 at 10:33, 'Paul Borman' via golang-nuts <
>> golan...@googlegroups.com> wrote:
>>
>>> That said, a map is represented by a single machine word (32 or 64
>>> bits).  I don't know of any modern architecture where reading a cache
>>> aligned word while it is being written will yield anything but the old or
>>> new value.
>>>
>>
>> This line of reasoning is what leads to trouble.  The atomicity problem
>> is not with the map reference, which will be either old or new, but with
>> all the memory writes to the actual hash table, which may be in any number
>> of intermediate states including "impossible" ones.  You must ensure that
>> the completion of those writes happens before another goroutine attempts to
>> read from the hashtable, and for that, you need some kind of barrier: a
>> channel, mutex, or atomic.Value.
>>
>
> Thank you for spelling this out! To put it into concrete terms, the hash
> table is likely spread over multiple cache lines and so if a goroutine
> executing on a processor other than the one which built the hash table was
> to load the cache line containing the new map reference and access it
> before loading the actual hash table cache lines that would spell disaster.
>
> I still kind of feel like a memory barrier after updating the map
> reference will be sufficient in practice; it doesn't seem likely that
> another processor would see the new map reference before the memory barrier
> caused the rest of the hash tables to also be visible? But I'm no x86
> export let alone ARM or less common architectures, so the boat is certainly
> starting to leak.
>
> Lesson learned; use barriers if sharing memory.
> -sqweek
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Having difficulty testing this "cleanly"

[Evan Digby]

Has anyone come across a good way, non-racy way to ensure that N tasks are 
guaranteed to be completed after a function is called? Essentially I have a 
“Close” function that must be guaranteed to block until all tasks are 
finished. Achieving this was pretty simple: wrap each task in an RLock, and 
then a Lock on close. 

Example: https://play.golang.org/p/7lhBPUhkUE

Now I want to write a solid test to guarantee Close will meet that 
requirement of all tasks must finish first for posterity. In that example, 
try commenting out the RLock/RUnlock on lines 25/26. You'll see that it no 
longer outputs many, if any, lines. I'm trying to prevent that from 
happening in the future by some cowboy refactor!

All of the ways I can come up with involve Sleeping or launching more tasks 
than I _think_ can be finished in time--obviously not good!

I feel like I must be missing some obvious way to test this and I'll end up 
feeling silly once someone replies with the solution. I'm okay with that!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Type Scope?

[aroman]

You need a pointer receiver on Set.  It copies the struct, sets the field 
on the copy, and throws it away:
https://play.golang.org/p/mqleaL6PU3

On Tuesday, September 13, 2016 at 11:41:46 AM UTC-7, Frank Davidson wrote:
>
> I think I'm having some type of mental lapse, but can anyone tell me why 
> this doesn't print out "It works!" nor give me an error? It simply prints 
> an empty string...
>
> https://play.golang.org/p/9oG0S8kfM3
>
> package main
>
> import "fmt"
>
> type fetcher struct { names string }
>
> func (f fetcher) Set(tname string) {
> f.names = tname
> }
>
> func main() {
> var me fetcher
> me.Set("It works!")
> fmt.Println(me.names)
> }
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Type Scope?

[Frank Davidson]

I think I'm having some type of mental lapse, but can anyone tell me why 
this doesn't print out "It works!" nor give me an error? It simply prints 
an empty string...

https://play.golang.org/p/9oG0S8kfM3

package main

import "fmt"

type fetcher struct { names string }

func (f fetcher) Set(tname string) {
f.names = tname
}

func main() {
var me fetcher
me.Set("It works!")
fmt.Println(me.names)
}

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] How Golang implements slice?

[Edward Muller]

Doh. I keep forgetting about a[low : high : max], but also want to point
out that if that if the slice is modified by the func beyond cap then the
backing array will be swapped out anyway and your slice header doesn't know
about the swap or the new cap.

On Tue, Sep 13, 2016 at 5:00 AM Val  wrote:

> Hello Ding
> Actually you don't need unsafe or reflect to access the 4th element :
> https://play.golang.org/p/wZAkKtAZei
> See section "Full slice expressions"
>  in spec about 3-index
> slicing like  a[low : high : max]
>
> You were correct in deducing that slices are descriptor values "passed
> around" by value, while the backing array contents are accessed by
> reference through the pointer contained in the slice descriptor.  This
> structure must be well understood by the programmer to avoid headaches.  My
> rules of thumb are :
> - avoid pointer to slice e.g.  *[]int
> - never call append without assigning back to the same variable
> - be careful with concurrency  e.g.  append is not safe to call
> concurrently
> - be careful about which piece of code "owns" the backing array.  This is
> important when several functions try to read, write, grow a slice.
> Problems may arise even without goroutines.
> - the result of append may use the same backing array as its argument, or
> not. You shouldn't rely on either way.
>
> Cheers
> Valentin
>
>
> On Tuesday, September 13, 2016 at 10:15:19 AM UTC+2, Fei Ding wrote:
>
>> Thanks, Edward, you make a good point.
>>
>> What's more, I wrote more code yesterday, and found that we can in fact
>> get the 4th element even if the descriptor tells us there are only 3, by
>> using package unsafe , and doing some
>> simple pointer calculation. It shows that function calls may modify the
>> slice in parameters under the cover, which is a bad thing if it is not on
>> purpose. I will keep this in mind.
>>
>> Thanks.
>>
>>
>>
>>
>> 2016-09-13 12:09 GMT+08:00 Edward Muller :
>>
>>>
>>>
>>> On Mon, Sep 12, 2016 at 8:17 PM Fei Ding  wrote:
>>>
 Hi guys:

 I met a problem when I try to understand slice when using as function
 parameters. After writing some code, I made a conclusion by myself, but
 need some real check and explanation here, with your help.

 As far as I know, Golang's function parameters are passed by value, and
 slice could be treated as a descriptor, which seems like *{pointer to
 first element of underlying array, length of element, capacity of array}*
 according to Go Slices: usage and internals
 . So, can I say:

 *When passing a slice as a function parameter, it is, in fact, passing
 a pointer, and two int values (or as a whole struct, or maybe more
 parameters to describe the slice, whatever).*

 To make it clear using code:

 func foo(s []T) {...}
 // can this function be treated as following 3-parameter function?
 func foo(p Pointer, length, capa int) {...}

 // Note: type is not the point here,
 //   and we just need 3 parameters, while there maybe more

 You may wonder why I have this question? Here is where everything
 started:

 func foo(s []int) {
  s := []int{1,2,3}
  foo(s)
  fmt.Println(s)
 }

 Everyone know it will print [1 2 3], not [1 2 3 100]. But what I am
 interested in, is *whether the function call does modify the memory*
 just after the tail of element 3, to an int of 100. If my assumption above
 is right, then the modification may happen.

 So, I made some experiments.

 package main

 import "fmt"
 import "strconv"

 type Node struct {
 val int
 left *Node
 right *Node
 }

 func (n Node) String () string {
 return strconv.Itoa(n.val)
 }

 func newNode(val int) *Node {
 return {val, nil, nil}
 }

 // Given a binary tree and a target sum, find all root-to-leaf paths
 // where each path's sum equals the given sum
 func bs(root *Node, path []*Node, now int, target int, ret *[][]*Node) {
 if root == nil {
 return
 }
 path = append(path, root)
 now += root.val
 if now == target && root.left == nil && root.right == nil {
 *ret = append(*ret, path)
 return
 }
 bs(root.left, path, now, target, ret)
 bs(root.right, path, now, target, ret)
 }

 func main() {
 // a simple tree like:
 //0
 //   / \
 //  1   2
 root := newNode(0)
 left := newNode(1)
 right := newNode(2)
 root.left = left
 root.right = right

 ret := [][]*Node{}
 bs(root, make([]*Node, 0, 10), 0, 1, )
 fmt.Println(ret)
 }


 As the code above, it is a function to find all root-to-leaf
 paths where each path's sum 

Re: [go-nuts] Multiple-reader single-writer map access - is this lockless workaround safe?

[sqweek E.]

On Tuesday, September 13, 2016 at 10:57:47 PM UTC+8, Alan Donovan wrote:
>
> On 13 September 2016 at 10:33, 'Paul Borman' via golang-nuts <
> golan...@googlegroups.com > wrote:
>
>> That said, a map is represented by a single machine word (32 or 64 
>> bits).  I don't know of any modern architecture where reading a cache 
>> aligned word while it is being written will yield anything but the old or 
>> new value.
>>
>
> This line of reasoning is what leads to trouble.  The atomicity problem is 
> not with the map reference, which will be either old or new, but with all 
> the memory writes to the actual hash table, which may be in any number of 
> intermediate states including "impossible" ones.  You must ensure that the 
> completion of those writes happens before another goroutine attempts to 
> read from the hashtable, and for that, you need some kind of barrier: a 
> channel, mutex, or atomic.Value.
>

Thank you for spelling this out! To put it into concrete terms, the hash 
table is likely spread over multiple cache lines and so if a goroutine 
executing on a processor other than the one which built the hash table was 
to load the cache line containing the new map reference and access it 
before loading the actual hash table cache lines that would spell disaster.

I still kind of feel like a memory barrier after updating the map reference 
will be sufficient in practice; it doesn't seem likely that another 
processor would see the new map reference before the memory barrier caused 
the rest of the hash tables to also be visible? But I'm no x86 export let 
alone ARM or less common architectures, so the boat is certainly starting 
to leak.

Lesson learned; use barriers if sharing memory.
-sqweek

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Cross-compilation of golang assembly via 'go tool asm'

[ironiridis]

On Tuesday, September 13, 2016 at 11:08:18 AM UTC-5, pcj...@gmail.com wrote:
>
> My next question is which tool is responsible for selecting the correct 
> sha1block_GOARCH.s? Is the the compiler, linker, or where?
>
 
See https://golang.org/pkg/go/build/ (under Build Constraints)
And (as of Go 1.7.1) https://golang.org/src/go/build/build.go#L1449 
goodOSArchFile()


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Multiple-reader single-writer map access - is this lockless workaround safe?

['Alan Donovan' via golang-nuts]

On 13 September 2016 at 12:36, sqweek E.  wrote:

> On Tuesday, September 13, 2016 at 9:48:06 AM UTC+8, Caleb Spare wrote:
>>
>> See https://software.intel.com/en-us/blogs/2013/01/06/benign-dat
>> a-races-what-could-possibly-go-wrong.
>>
>
> I've read this article before and the second reading hasn't done much to
> improve my impression. The premise in the first example, that multiple
> threads updating the same memory location in a non-atomic fashion
> constitutes "innocent" code, is about as strawman as they come. Forget the
> data race, there's an obvious race condition here! And unless C++ atomic
> operations are wildly different from those I have experience with (java
> mainly), the suggested "correct" code fixes only the data race and not the
> race condition.
>

I don't think it's a straw man.  Many novice programmers wrongly assume
that the behavior of a concurrent program can be understood as an arbitrary
interleaving of the statements of the threads of that program, and this is
almost true for a few languages, such as Python.  The discovery that it
cannot may be profoundly surprising.


And problems emerging from all of these get blamed on the programmer's
> "racy code" instead of a broken compiler?
>

The authors of the Go spec tend to take a more user-friendly line than the
C standard.  For example, the Go memory model says that the value observed
by a racy read of a single-word variable is not arbitrary, but must equal
one of the values previously or concurrently stored there. In contrast, C
doesn't define this behavior at all.  Whether C-based compilers such as
gccgo or llgo actually implement this behavior is an interesting question.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Multiple-reader single-writer map access - is this lockless workaround safe?

[sqweek E.]

On Tuesday, September 13, 2016 at 9:48:06 AM UTC+8, Caleb Spare wrote:
>
> See 
> https://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong.
>  
>
>

I've read this article before and the second reading hasn't done much to 
improve my impression. The premise in the first example, that multiple 
threads updating the same memory location in a non-atomic fashion 
constitutes "innocent" code, is about as strawman as they come. Forget the 
data race, there's an obvious race condition here! And unless C++ atomic 
operations are wildly different from those I have experience with (java 
mainly), the suggested "correct" code fixes only the data race and not the 
race condition.

The remaining examples sound indistinguishable from compiler bugs. You have 
multiple threads concurrently accessing a 'stop' variable, and yet the 
compiler believes that "the stop variable can't be accessed concurrently by 
other threads"? Might want to consider some escape analysis before making 
that assumption...

You define a function-local variable intended to hold a copy of some shared 
memory, and the compiler decides to read the shared memory twice? Thanks 
buddy!

The last one takes the cake - you read from a memory location and the 
compiler TURNS IT INTO A WRITE??? And problems emerging from all of these 
get blamed on the programmer's "racy code" instead of a broken compiler?

W. T. F.
 

> Most Go programmers consider programs with data races to be logically 
> incorrect / broken. 
>

Maybe there are good reasons to consider them such, but this article has 
not convinced me of any.
-sqweek

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Max age for TCP connection

[C Banning]

What is your http.Client.Timeout value?

On Tuesday, September 13, 2016 at 8:57:45 AM UTC-6, david...@ft.com wrote:
>
> Hi,
>
> We've got a HTTP client connecting to an active-passive cluster setup, 
> controlled through a traffic managed DNS entry and a good-to-go signal on 
> each cluster.  When the good-to-go fails, the DNS switches to point at the 
> passive cluster.  The client in question has no knowledge of whether a 
> failover has occurred, it's just hitting an endpoint on the traffic managed 
> address.
>
> The client is resolving the DNS correctly and opening a TCP connection. 
>  However, when the failover occurs, the TCP connection to the old cluster 
> stays active because of Go's persistent connections.  The connection is not 
> idle (it still makes scheduled requests to the endpoint via its IP address) 
> and so is not closed when we close all idle connections.
>
> Is there some way to set a maximum age for the TCP connection?
>
> --
>
> *This email was sent by a company owned by Financial Times Group Limited 
> ("FT Group "), 
> registered office at Number One Southwark Bridge, London SE1 9HL.  
> Registered in England and Wales with company number 879531. This e-mail may 
> contain confidential information. If you are not the intended recipient, 
> please notify the sender immediately, delete all copies and do not 
> distribute it further.  It could also contain personal views which are not 
> necessarily those of the FT Group.  We may monitor outgoing or 
> incoming emails as permitted by law.*
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Static race detector?

[josvazg]

Thanks for all links and answers!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] What is the correct usage for the -msan flag?

[Ian Lance Taylor]

On Tue, Sep 13, 2016 at 5:03 AM, Markus Zimmermann  wrote:
> On Friday, September 9, 2016 at 1:51:49 PM UTC+2, Ian Lance Taylor wrote:
>>
>> > I have the following questions concerning the usage of the -msan flag:
>> > a.) Are these the preferred arguments to use the -msan flag?
>>
>> Looks OK to me.
>
> I am wondering if they are really needed, and if yes, why are they not
> included at compile time by the Go compiler?

The Go tool will add the -fsanitize=memory option, so you shouldn't
need to add that yourself.  It won't add the other options, though.

> Also, would it make sense to
> add -fsanitize-memory-track-origins as -msan-track-origins flag to Go build?

I think not, because I think that is a choice for the user to make.


>> > d.) What are the official minimum requirements for the -msan flag? (LLVM
>> > >=
>> > 3.8? ...?)
>>
>> Yes, on GNU/Linux you need at least LLVM 3.8.  I don't know of any
>> other requirements.
>
> I am wondering if the version requirement should be added to the help text
> of -msan? It currently states "enable interoperation with memory sanitizer.
> Supported only on linux/amd64, and only with Clang/LLVM as the host C
> compiler." I am also wondering why these conditions (as stated in
> https://golang.org/misc/cgo/testsanitizers/test.bash ) are not checked
> during compile time? Would that be a good Go addition?

I don't think it makes sense to add specific version numbers to the
-msan help text.  The specific details can change with changing Linux
versions, and really the compatibility has nothing to do with Go.
Perhaps it would make sense to say something like "requires a version
of LLVM that supports -fsanitize=memory on the version of GNU/Linux
that the program runs on."  I don't know how much that would help
people, though.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Cross-compilation of golang assembly via 'go tool asm'

[Aram Hăvărneanu]

On Tue, Sep 13, 2016 at 6:19 AM,   wrote:
> My interpretation of golang's assembly is that it represents an intermediate
> pseudo-language that is transformed via "instruction selection" to a
> concrete form.

Correct, but this pseudo-language is not portable between architectures.

> Therefore, is it possible to re-write the above to a cross-compilable form?

The assembly in the Go distribution is already cross-compilable, but
looking at your example you are not concerned whether the assembly is
cross-compilable or not (that depends on the toolchain, and not on the
assembly code), but you want the assembly to be portable between
architectures.

Apart from the most trivial of cases, this is not possible. Different
targets use different instructions and different registers.

The Go assembler uses the same *syntax* on every architecture, so it
can be parsed and processed by a single program, however the assembly
code required for each target is very different.

-- 
Aram Hăvărneanu

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Multiple-reader single-writer map access - is this lockless workaround safe?

['Paul Borman' via golang-nuts]

Yes, that is very true, though in the OP's case, the memory barrier can
easily be done by the writer without affecting the reader (simply by
switching the write to the channel with the assignment of x.myMap).
Granted, the reader might have stale information, but I believe that the
compiler currently will only cache the pointer to the map with relative
locality, so for practical (but not guaranteed) purposes, it will not
appear to be a problem.

But I will reiterate what I said:  *The Go memory model does not guarantee
this, however, so while it might work for you now on your current target,
that is a side effect of an implementation and architecture detail.*



On Tue, Sep 13, 2016 at 7:57 AM, Alan Donovan  wrote:

> On 13 September 2016 at 10:33, 'Paul Borman' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
>> That said, a map is represented by a single machine word (32 or 64
>> bits).  I don't know of any modern architecture where reading a cache
>> aligned word while it is being written will yield anything but the old or
>> new value.
>>
>
> This line of reasoning is what leads to trouble.  The atomicity problem is
> not with the map reference, which will be either old or new, but with all
> the memory writes to the actual hash table, which may be in any number of
> intermediate states including "impossible" ones.  You must ensure that the
> completion of those writes happens before another goroutine attempts to
> read from the hashtable, and for that, you need some kind of barrier: a
> channel, mutex, or atomic.Value.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Max age for TCP connection

[david . smith]

Hi,

We've got a HTTP client connecting to an active-passive cluster setup, 
controlled through a traffic managed DNS entry and a good-to-go signal on 
each cluster.  When the good-to-go fails, the DNS switches to point at the 
passive cluster.  The client in question has no knowledge of whether a 
failover has occurred, it's just hitting an endpoint on the traffic managed 
address.

The client is resolving the DNS correctly and opening a TCP connection. 
 However, when the failover occurs, the TCP connection to the old cluster 
stays active because of Go's persistent connections.  The connection is not 
idle (it still makes scheduled requests to the endpoint via its IP address) 
and so is not closed when we close all idle connections.

Is there some way to set a maximum age for the TCP connection?

-- 

--

*This email was sent by a company owned by Financial Times Group Limited 
("FT Group "), 
registered office at Number One Southwark Bridge, London SE1 9HL.  
Registered in England and Wales with company number 879531. This e-mail may 
contain confidential information. If you are not the intended recipient, 
please notify the sender immediately, delete all copies and do not 
distribute it further.  It could also contain personal views which are not 
necessarily those of the FT Group.  We may monitor outgoing or 
incoming emails as permitted by law.*

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Multiple-reader single-writer map access - is this lockless workaround safe?

['Alan Donovan' via golang-nuts]

On 13 September 2016 at 10:33, 'Paul Borman' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> That said, a map is represented by a single machine word (32 or 64 bits).
> I don't know of any modern architecture where reading a cache aligned word
> while it is being written will yield anything but the old or new value.
>

This line of reasoning is what leads to trouble.  The atomicity problem is
not with the map reference, which will be either old or new, but with all
the memory writes to the actual hash table, which may be in any number of
intermediate states including "impossible" ones.  You must ensure that the
completion of those writes happens before another goroutine attempts to
read from the hashtable, and for that, you need some kind of barrier: a
channel, mutex, or atomic.Value.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Multiple-reader single-writer map access - is this lockless workaround safe?

['Paul Borman' via golang-nuts]

I think it is pretty common to make your map/list "read only" and to update
it by a replace.  That does not get you away from the need for a mutex,
however.   There is still a reader and writer:

Reader:

v := m[key]


Writer:

m = newMap


Those two operations are not safe together.  You still need a mutex:

mu.Lock()
mc := m
mu.Unlock()
v := mc[key]

mu.Lock()
m = newMap
mu.Unlock()


Of course, if you are not going to fetch multiple values from mc, it
degenerates to:

mu.Lock()
v := m[key]
mu.Unlock()


Which is what you are trying to avoid.

That said, a map is represented by a single machine word (32 or 64 bits).
I don't know of any modern architecture where reading a cache aligned word
while it is being written will yield anything but the old or new value.
The Go memory model does not guarantee this, however, so while it might
work for you now on your current target, that is a side effect of an
implementation and architecture detail.

There are some architectures that allow non-aligned reads/writes where a
machine word could span cache lines, but that is very expensive and I do
not believe Go ever generates code like that, though I guess it could be
possible.

-Paul

On Mon, Sep 12, 2016 at 6:47 PM, Caleb Spare  wrote:

> If you have a data race, you lose a lot of guarantees about the
> correct operation of your program. It may work today but fail
> occasionally, or a future version of the compiler may break it very
> badly.
>
> See https://software.intel.com/en-us/blogs/2013/01/06/benign-
> data-races-what-could-possibly-go-wrong.
>
> Most Go programmers consider programs with data races to be logically
> incorrect / broken.
>
> -Caleb
>
> On Mon, Sep 12, 2016 at 6:41 PM, sqweek E.  wrote:
> > Thanks all. "Plain assignment" is not quite the full story actually - as
> I
> > mentioned originally there's also a channel in use to wake up the painter
> > thread. So in terms of memory model what I'm doing is more like:
> >
> > x.myMap = newMap
> > x.refresh <- nil // i don't think it matters what value is sent
> across
> > the channel?
> >
> > Also I don't mind a bit of racy non-determinism. If the painter thread
> > happens to be actively painting when I do this update I don't care if it
> > notices the newMap immediately or not, just as long as it sees the update
> > when painting the next frame. That will be after it has drained the
> refresh
> > channel, so should be guaranteed by the memory model if I've understood
> > correctly.
> >
> > For the general case sync.Value sounds appropriate. It means touching all
> > the reader code aswell, but in a fairly trivial way.
> >
> > Thanks again!
> > -sqweek
> >
> > On Tuesday, September 13, 2016 at 12:24:09 AM UTC+8, Peter Bourgon wrote:
> >>
> >> All memory operations are unsafe for concurrent goroutines unless
> >> explicitly noted otherwise. In practice, if you have multiple
> >> goroutines accessing the same memory, you need to protect it with the
> >> primitives available to you in package sync or sync/atomic. Please
> >> review https://golang.org/ref/mem for more details.
> >>
> >> Many Go programmers opt to sidestep these sorts of problems by using
> >> channels to orchestrate behavior and/or the flow of data between
> >> goroutines — i.e. not communicating by sharing memory, but rather
> >> sharing memory by communicating.
> >>
> >>
> >> On Mon, Sep 12, 2016 at 6:04 PM, sqweek E.  wrote:
> >> > Yes, through plain assignment. What problems arise from that?
> >> >
> >> > On Monday, September 12, 2016 at 11:21:05 PM UTC+8, Peter Bourgon
> wrote:
> >> >>
> >> >> How are you replacing the reference to the original map? If you're
> >> >> just using plain assignment e.g. x.myMap = newMap then that's not
> >> >> safe. You could use sync/atomic.Value if you always made sure to do
> >> >> map reads via the Load method.
> >> >>
> >> >> On Mon, Sep 12, 2016 at 5:04 PM, sqweek E.  wrote:
> >> >> > instead of updating the maps in place, they (a) take a
> >> >> > copy of the current map (b) update the copy and (c) replace the
> >> >> > reference to
> >> >> > the original map (held in a struct field) with the updated copy.
> >> >>
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "golang-nuts" group.
> >> > To unsubscribe from this group and stop receiving emails from it, send
> >> > an
> >> > email to golang-nuts...@googlegroups.com.
> >> > For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to golang-nuts+unsubscr...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe 

Re: [go-nuts] [ANN] google.golang.org/cloud import paths removed

[Steven Hartland]

The links on the docs for this are currently broken e.g.
https://cloud.google.com/go/apis points to 
https://godoc.org/google.golang.org/cloud


On 13/09/2016 12:59, jba via golang-nuts wrote:
As was announced six weeks ago, the Google Cloud API clients, formerly 
residing at google.golang.org/cloud/..., have changed import paths to 
cloud.google.com/go/ The old import paths have now been removed. 
For details, 
see https://groups.google.com/forum/#!topic/google-api-go-announce/82-qWvuZzrI. 



--
You received this message because you are subscribed to the Google 
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to golang-nuts+unsubscr...@googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Why is there no " declared and not used" compile error?

['Alan Donovan' via golang-nuts]

On 13 September 2016 at 08:22, Markus Zimmermann  wrote:

> On Monday, September 12, 2016 at 3:41:35 PM UTC+2, adon...@google.com
> wrote:
>>
>> unused constants and types cost nothing at run time.  It's not that
>> simple, of course, because constant and type declarations may be the only
>> uses of an import of a package whose initialization has some cost; but this
>> is quite marginal.
>>
>
> The const declaration of my example is inside an unexported function body.
> It can not be accessed by an import. So, it should really matter only at
> compile time, right?
>

Right.  The import case I was referring to is this:

import "p"
func main() {
  const unused = p.K
}

The unused constant causes p to be an (unnecessary) dependency, which in
turn causes p.init to be executed.

However, I am referring to the definition of https://golang.org/doc/faq#
> unused_variables_and_imports which states "... Go refuses to compile
> programs with unused variables or imports, trading short-term convenience
> for long-term build speed and program clarity." Which holds true in the
> same way for an unused const as it does for an unused variable. However,
> even though it produces the same results, i.e. a new declaration which
> decreases the program clarity, only the variable declaration is marked.
>

You are right that the motivation given in the FAQ should apply equally to
constants and types.

 Additionally, if an unused constant is compiled into the binary it does
> also matter to the binary size, even just a little bit.


An unused constant is unlikely to affect the size of the final executable
since in a typical compiler generates code or data for a constant only when
it is used. It will be present in the intermediate .a files though.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Another divide by zero (float) question

['Paul Borman' via golang-nuts]

float32(0) is a constant, a typed constant.  The following statement is
valid:

const z = float32(0)

So it is following the part of the spec that Ian pointed out.

-Paul

On Mon, Sep 12, 2016 at 5:15 PM,  wrote:

> I see - so if I understood correctly the spec for constants disallowing
> divide by zero has accidentally been applied to divide by zero for float32s
>
> Does this mean there will be a fix at some distant point ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Why is there no " declared and not used" compile error?

[Markus Zimmermann]

On Monday, September 12, 2016 at 3:21:50 PM UTC+2, Sjon Kosse wrote:
>
> Hello,
>
> Const values can be accesed still by reflection in a reasonable manner. 
> This question has come by quite a few times before on the mailing list, 
> better explanations should be easy to find.
>

I do not think that this is true. Wouldn't this already break the first 
reflection law? How can you examine something with the reflection package 
without mentioning it in the source code? Where would the interface value 
come from? Additionally, my example does not export anything, it is a 
constant inside an unexported function.


Cheers,
Markus

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Defective information output??? (To Developers)

[Jan Mercl]

On Tue, Sep 13, 2016 at 2:19 PM Александр Петров 
wrote:

> Consult me. I believe that the displayed information is incorrect and
should be changed in future versions of the language.
> What I see, it is of little help in debugging.

https://play.golang.org/p/jQJJoknb3v


-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Defective information output??? (To Developers)

[Rob Pike]

The displayed information is correct as specified. Use a different format
if you want different output. In this example, %q is a good choice.

https://play.golang.org/p/Wyxv8md1qu

-rob



On Tue, Sep 13, 2016 at 4:35 PM, Александр Петров 
wrote:

> Hello.
>
> https://play.golang.org/p/4IlgcBIbxk
>
> Consult me. I believe that the displayed information is incorrect and
> should be changed in future versions of the language.
> What I see, it is of little help in debugging.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Why is there no " declared and not used" compile error?

[Markus Zimmermann]

On Monday, September 12, 2016 at 3:41:35 PM UTC+2, adon...@google.com wrote:
>
> On Monday, 12 September 2016 08:57:15 UTC-4, Markus Zimmermann wrote:
>>
>> Hi gophers!
>>
>> I am wondering why we have a "declared and not used" compile error for 
>> variables [https://play.golang.org/p/KLzHVO5L7q] but not for constants [
>> https://play.golang.org/p/JG9dSa_VKg]? I am sure there is some rational 
>> decision behind this that I have missed, but my Google-fu is just not 
>> strong enough to find an answer.
>>
>
> I suspect the answer is: because unused constants and types cost nothing 
> at run time.  It's not that simple, of course, because constant and type 
> declarations may be the only uses of an import of a package whose 
> initialization has some cost; but this is quite marginal.
>

The const declaration of my example is inside an unexported function body. 
It can not be accessed by an import. So, it should really matter only at 
compile time, right? However, I am referring to the definition of 
https://golang.org/doc/faq#unused_variables_and_imports which states "... 
Go refuses to compile programs with unused variables or imports, trading 
short-term convenience for long-term build speed and program clarity." 
Which holds true in the same way for an unused const as it does for an 
unused variable. However, even though it produces the same results, i.e. a 
new declaration which decreases the program clarity, only the variable 
declaration is marked. Additionally, if an unused constant is compiled into 
the binary it does also matter to the binary size, even just a little bit.

Cheers,
Markus

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Defective information output??? (To Developers)

[Александр Петров]

Hello.

https://play.golang.org/p/4IlgcBIbxk

Consult me. I believe that the displayed information is incorrect and 
should be changed in future versions of the language.
What I see, it is of little help in debugging.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] What is the correct usage for the -msan flag?

[Markus Zimmermann]

On Friday, September 9, 2016 at 1:51:49 PM UTC+2, Ian Lance Taylor wrote:

> > I have the following questions concerning the usage of the -msan flag: 
> > a.) Are these the preferred arguments to use the -msan flag? 
>
> Looks OK to me. 
>

I am wondering if they are really needed, and if yes, why are they not 
included at compile time by the Go compiler? Also, would it make sense to 
add -fsanitize-memory-track-origins as -msan-track-origins flag to Go build?

> b.) Why are there no line numbers in the traces? 
>
> I don't know.  Try adding -g to CGO_CPPFLAGS to see if it makes a 
> difference. 
>
> > c.) Any advice on how I can debug such problems? 
>
> If -g doesn't help then I think you'll have to use objdump or gdb or 
> lldb to look at the code at the failing address.  Those tools should 
> at least let you map that address back to a source line.  Note that 
> the problem does not seem to be in Go code, which of course is 
> unsurprising. 
>

Unfortunately -g does not make a difference. However, you are right that 
this is not a Go problem and I will create a bug report in the LLVM tracker.

> d.) What are the official minimum requirements for the -msan flag? (LLVM 
> >= 
> > 3.8? ...?) 
>
> Yes, on GNU/Linux you need at least LLVM 3.8.  I don't know of any 
> other requirements. 
>

I am wondering if the version requirement should be added to the help text 
of -msan? It currently states "enable interoperation with memory sanitizer. 
Supported only on linux/amd64, and only with Clang/LLVM as the host C 
compiler." I am also wondering why these conditions (as stated in 
https://golang.org/misc/cgo/testsanitizers/test.bash ) are not checked 
during compile time? Would that be a good Go addition?

Thanks for your answers. They are really helpful.

Cheers,
Markus

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] How Golang implements slice?

[Val]

Hello Ding
Actually you don't need unsafe or reflect to access the 4th element :  
https://play.golang.org/p/wZAkKtAZei
See section "Full slice expressions" 
 in spec about 3-index 
slicing like  a[low : high : max]

You were correct in deducing that slices are descriptor values "passed 
around" by value, while the backing array contents are accessed by 
reference through the pointer contained in the slice descriptor.  This 
structure must be well understood by the programmer to avoid headaches.  My 
rules of thumb are :
- avoid pointer to slice e.g.  *[]int
- never call append without assigning back to the same variable
- be careful with concurrency  e.g.  append is not safe to call concurrently
- be careful about which piece of code "owns" the backing array.  This is 
important when several functions try to read, write, grow a slice.  
Problems may arise even without goroutines.
- the result of append may use the same backing array as its argument, or 
not. You shouldn't rely on either way.

Cheers
Valentin

On Tuesday, September 13, 2016 at 10:15:19 AM UTC+2, Fei Ding wrote:
>
> Thanks, Edward, you make a good point. 
>
> What's more, I wrote more code yesterday, and found that we can in fact 
> get the 4th element even if the descriptor tells us there are only 3, by 
> using package unsafe , and doing some 
> simple pointer calculation. It shows that function calls may modify the 
> slice in parameters under the cover, which is a bad thing if it is not on 
> purpose. I will keep this in mind.
>
> Thanks.
>
>
>
>
> 2016-09-13 12:09 GMT+08:00 Edward Muller  >:
>
>>
>>
>> On Mon, Sep 12, 2016 at 8:17 PM Fei Ding  
>> wrote:
>>
>>> Hi guys:
>>>
>>> I met a problem when I try to understand slice when using as function 
>>> parameters. After writing some code, I made a conclusion by myself, but 
>>> need some real check and explanation here, with your help.
>>>
>>> As far as I know, Golang's function parameters are passed by value, and 
>>> slice could be treated as a descriptor, which seems like *{pointer to 
>>> first element of underlying array, length of element, capacity of array}* 
>>> according to Go Slices: usage and internals 
>>> . So, can I say:
>>>
>>> *When passing a slice as a function parameter, it is, in fact, passing a 
>>> pointer, and two int values (or as a whole struct, or maybe more parameters 
>>> to describe the slice, whatever).*
>>>
>>> To make it clear using code:
>>>
>>> func foo(s []T) {...}
>>> // can this function be treated as following 3-parameter function? 
>>> func foo(p Pointer, length, capa int) {...}
>>>
>>> // Note: type is not the point here,
>>> //   and we just need 3 parameters, while there maybe more
>>>
>>> You may wonder why I have this question? Here is where everything 
>>> started:
>>>
>>> func foo(s []int) {
>>>  s := []int{1,2,3}
>>>  foo(s)
>>>  fmt.Println(s)
>>> }
>>>
>>> Everyone know it will print [1 2 3], not [1 2 3 100]. But what I am 
>>> interested in, is *whether the function call does modify the memory* 
>>> just after the tail of element 3, to an int of 100. If my assumption above 
>>> is right, then the modification may happen.
>>>
>>> So, I made some experiments.
>>>
>>> package main
>>>
>>> import "fmt"
>>> import "strconv"
>>>
>>> type Node struct {
>>> val int
>>> left *Node
>>> right *Node
>>> }
>>>
>>> func (n Node) String () string {
>>> return strconv.Itoa(n.val)
>>> }
>>>
>>> func newNode(val int) *Node {
>>> return {val, nil, nil}
>>> }
>>>
>>> // Given a binary tree and a target sum, find all root-to-leaf paths 
>>> // where each path's sum equals the given sum
>>> func bs(root *Node, path []*Node, now int, target int, ret *[][]*Node) {
>>> if root == nil {
>>> return
>>> }
>>> path = append(path, root)
>>> now += root.val
>>> if now == target && root.left == nil && root.right == nil {
>>> *ret = append(*ret, path)
>>> return
>>> }
>>> bs(root.left, path, now, target, ret)
>>> bs(root.right, path, now, target, ret)
>>> }
>>>
>>> func main() {
>>> // a simple tree like:
>>> //0
>>> //   / \
>>> //  1   2
>>> root := newNode(0)
>>> left := newNode(1)
>>> right := newNode(2)
>>> root.left = left
>>> root.right = right
>>>
>>> ret := [][]*Node{}
>>> bs(root, make([]*Node, 0, 10), 0, 1, )
>>> fmt.Println(ret)
>>> }
>>>
>>>
>>> As the code above, it is a function to find all root-to-leaf paths where 
>>> each path's sum equals the given sum in a binary tree, and, I make a simple 
>>> test case which there are only 3 nodes: one root, two children, with values 
>>> of 0, 1 and 2. 
>>>
>>> Say, I want to find the paths of sum of 1. So, I call this function as:
>>>
>>> bs(root, make([]*Node, 0, 10), 0, 1, )
>>>
>>> It is petty common to make a guess that, this call will give us a final 
>>> result of [0, 1], which is obviously the correct answer, however, it 

[go-nuts] [ANN] google.golang.org/cloud import paths removed

[jba via golang-nuts]

As was announced six weeks ago, the Google Cloud API clients, formerly 
residing at google.golang.org/cloud/..., have changed import paths to 
cloud.google.com/go/ The old import paths have now been removed. For 
details, 
see https://groups.google.com/forum/#!topic/google-api-go-announce/82-qWvuZzrI.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: [ANN] A Go based map reduce system (second attempt)

[Daniel Whitenack]

Henrik and David,

Yes, this is precisely what pachyderm is trying to accomplish (along with a 
built in data versioning system).  It's pretty wonderful, and I recommend 
running through their quick start guide:

http://pachyderm.readthedocs.io/en/latest/

Daniel

On Monday, September 12, 2016 at 1:56:24 PM UTC-4, Henrik Johansson wrote:
>
> Perhaps something like this?
>
> https://github.com/pachyderm/chess
>
>
>
> mån 12 sep. 2016 kl 18:32 skrev David Crawshaw  >:
>
>> The map-reduce systems I've worked with in the past bundle up the map
>> and reduce functions in a binary, distribute it, and then communicate
>> with it by an IPC or RPC system. This lets them take advantage of the
>> operating system to control resource allocation. (Given how popular
>> containers are becoming, I expect someone will do it with Linux
>> containers to provide even better isolation.) I suspect this is a more
>> flexible approach than using plugins.
>>
>> On Mon, Sep 12, 2016 at 2:38 AM, Raj > > wrote:
>> >>> However, it is still limited by the fact that Go code can not be sent 
>> and
>> >>> executed remotely.
>> >
>> > I am not sure if this makes the situation better for this particular use
>> > case, but I see that David Crawshaw is working on plugin mode for Go. It
>> > looks like it will be part of Go1.8.
>> >
>> > --
>> > You received this message because you are subscribed to the Google 
>> Groups
>> > "golang-nuts" group.
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> an
>> > email to golang-nuts...@googlegroups.com .
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: memory leak with websocket

[Dave Cheney]

I'm glad you found the cause of the leak. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: memory leak with websocket service based on go-socket.io

[aiden0xz]

Finally I fixed the problem.

I use the https://github.com/auth0/go-jwt-middleware to auth with JWT. I 
also open the KeepContext of gorilla mux router to true for that can get 
the jwt token of  request from the goriila context. When I get the jwt 
token, will clear the request context (via context.Clear(r))  immediately. 
So far, all is work correctly. *But when the go-jwt-middleware auth failed 
there is no chance to clear the request context because KeepContext of mux 
has been to true.* This is the root cause. 



在 2016年9月6日星期二 UTC+8上午10:52:35，aide...@gmail.com写道：
>
> We have a simple websocket service based on go-socket.io 
>  and have a memory leak 
> problem.  We found the memory usage is always increase, the idle heap is 
> not return back to the OS. The heap pprof show that the runtime.goexit took 
> much memory,  seems  that the memory not release after the goroutine exit. 
>
> 1. *heap callgraph*
>
>
> 
>
>
>
>
> 2. *gc debug log*
>
> gc 440 @51045.897s 0%: 0.034+4182+0.96 ms clock, 0.13+0/4182/12246+3.8 ms 
> cpu, 4304->4309->4143 MB, 8266 MB goal, 4 P
> scvg340: inuse: 4404, idle: 15, sys: 4419, released: 0, consumed: 4419 (MB)
> GC forced
> gc 441 @51170.096s 0%: 3.7+4355+1.4 ms clock, 14+2.9/4357/12795+5.8 ms 
> cpu, 4317->4323->4158 MB, 8287 MB goal, 4 P
> GC forced
> gc 442 @51294.460s 0%: 0.034+3987+1.2 ms clock, 0.13+1.5/3987/11701+4.9 ms 
> cpu, 4336->4341->4169 MB, 8316 MB goal, 4 P
> scvg341: inuse: 4318, idle: 133, sys: 4451, released: 0, consumed: 4451 
> (MB)
> GC forced
> gc 443 @51418.451s 0%: 0.36+3925+0.99 ms clock, 1.4+4.0/3925/11554+3.9 ms 
> cpu, 4350->4356->4182 MB, 8338 MB goal, 4 P
> scvg342: inuse: 4363, idle: 103, sys: 4466, released: 0, consumed: 4466 
> (MB)
> GC forced
> gc 444 @51542.394s 0%: 0.042+3986+1.6 ms clock, 0.16+0/3981/11757+6.5 ms 
> cpu, 4361->4367->4194 MB, 8365 MB goal, 4 P
> scvg343: inuse: 4404, idle: 74, sys: 4478, released: 0, consumed: 4478 (MB)
> GC forced
> gc 445 @51666.384s 0%: 3.4+3987+1.4 ms clock, 13+2.5/3986/11747+5.7 ms 
> cpu, 4375->4382->4208 MB, 8388 MB goal, 4 P
> scvg344: inuse: 4454, idle: 39, sys: 4493, released: 0, consumed: 4493 (MB)
> GC forced
> gc 446 @51790.379s 0%: 0.055+4147+1.5 ms clock, 0.22+0/4139/12125+6.2 ms 
> cpu, 4396->4402->4220 MB, 8416 MB goal, 4 P
> scvg345: inuse: 4509, idle: 5, sys: 4514, released: 0, consumed: 4514 (MB)
> GC forced
> gc 447 @51914.542s 0%: 0.052+4205+2.1 ms clock, 0.21+1.5/4199/12348+8.5 ms 
> cpu, 4413->4420->4234 MB, 8441 MB goal, 4 P
> GC forced
> gc 448 @52038.752s 0%: 2.7+4517+1.8 ms clock, 11+2.3/4517/13245+7.2 ms 
> cpu, 4428->4436->4247 MB, 8469 MB goal, 4 P
> scvg346: inuse: 4406, idle: 142, sys: 4548, released: 0, consumed: 4548 
> (MB)
> GC forced
> gc 449 @52163.276s 0%: 0.033+4206+1.3 ms clock, 0.13+0/4206/12306+5.3 ms 
> cpu, 4442->4449->4259 MB, 8495 MB goal, 4 P
> scvg347: inuse: 4452, idle: 109, sys: 4561, released: 0, consumed: 4561 
> (MB)
> GC forced
> gc 450 @52287.491s 0%: 0.044+4262+2.0 ms clock, 0.17+0/4261/12565+8.2 ms 
> cpu, 4452->4459->4272 MB, 8519 MB goal, 4 P
> scvg348: inuse: 4498, idle: 74, sys: 4572, released: 0, consumed: 4572 (MB)
> GC forced
> gc 451 @52411.769s 0%: 0.028+4012+2.0 ms clock, 0.11+0.066/3992/11762+8.0 
> ms cpu, 4471->4477->4285 MB, 8544 MB goal, 4 P
> scvg349: inuse: 4550, idle: 40, sys: 4590, released: 0, consumed: 4590 (MB)
>
> *3. example code*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *   server.On("error", func(so socketio.Socket, err error) {  
>   logger := SocketLogger(so)logger.Error("socket 
> connect error")})server.On("connection", func(so 
> socketio.Socket) {var (uid   
> stringexist bool)
> logger := SocketLogger(so)claim := 
> (context.Get(so.Request(), "user")).(*jwt.Token).Claims// 
> after get the claims, should clear the request context
> context.Clear(so.Request())var rawUID interface{}  
>   if user, ok := claim.(jwt.MapClaims); ok {if 
> rawUID, ok = user[setting.JwtUserClaimField]; !ok {
> logger.Error("invalid user claim")
> so.Emit("disconnect", "invalid user claim")}
> } else {logger.Errorf("invalid jwt claim 
> %s", claim)so.Emit("disconnect", "invalid user 
> claim")}if uid, exist = rawUID.(string); 
> exist {// Multi connection for same user will be 
> join to the same roomso.Join(uid)  
>   // root for broadcast all user
> 

Re: [go-nuts] parsing millisecond coming after colon

[Geetha Adinarayan]

Thank you. This is helpful. I am going to replace : with . first.

On Tuesday, September 13, 2016 at 9:09:31 AM UTC+5:30, Caleb Spare wrote:
>
> As Dave said, you should check the error, but unfortunately I think 
> you'll find that time.Parse doesn't handle fractional seconds that use 
> a character other than . as a decimal separator. 
>
> One possible workaround would be to replace all the : with . first: 
>
> https://play.golang.org/p/h_IMQxtoVI 
>
> -Caleb 
>
> On Mon, Sep 12, 2016 at 1:17 PM, Geetha Adinarayan 
>  wrote: 
> > Hi, 
> > 
> > I am trying to parse time coming in "6/9/15 15:54:04:393 CST" format. 
> Note : 
> > before millisecond. When there is dot before millisecond, time.parse is 
> > working fine.. but when there is : before millisecond, time.parse does 
> not 
> > work. am I missing anything? 
> > 
> > I tried the following 
> > 
> > package main 
> > 
> > import ( 
> > "fmt" 
> > "time" 
> > 
> > ) 
> > 
> > func main() { 
> >   const inputTime = "1/2/06 15:04:05:000 MST" 
> >   t, _ := time.Parse(inputTime, "6/9/15 15:54:04:393 CST") 
> >   fmt.Println(t) 
> >   fmt.Println(t.Format("2006-01-02T15:04:05.000-0700")) 
> > } 
> > 
> > but I get the following 
> > 
> > 0001-01-01 00:00:00 + UTC 
> > 0001-01-01T00:00:00.000+ 
> > 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups 
> > "golang-nuts" group. 
> > To unsubscribe from this group and stop receiving emails from it, send 
> an 
> > email to golang-nuts...@googlegroups.com . 
> > For more options, visit https://groups.google.com/d/optout. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] How Golang implements slice?

[Fei Ding]

Thanks, Edward, you make a good point.

What's more, I wrote more code yesterday, and found that we can in fact get
the 4th element even if the descriptor tells us there are only 3, by
using package
unsafe , and doing some simple pointer
calculation. It shows that function calls may modify the slice in
parameters under the cover, which is a bad thing if it is not on purpose. I
will keep this in mind.

Thanks.




2016-09-13 12:09 GMT+08:00 Edward Muller :

>
>
> On Mon, Sep 12, 2016 at 8:17 PM Fei Ding  wrote:
>
>> Hi guys:
>>
>> I met a problem when I try to understand slice when using as function
>> parameters. After writing some code, I made a conclusion by myself, but
>> need some real check and explanation here, with your help.
>>
>> As far as I know, Golang's function parameters are passed by value, and
>> slice could be treated as a descriptor, which seems like *{pointer to
>> first element of underlying array, length of element, capacity of array}*
>> according to Go Slices: usage and internals
>> . So, can I say:
>>
>> *When passing a slice as a function parameter, it is, in fact, passing a
>> pointer, and two int values (or as a whole struct, or maybe more parameters
>> to describe the slice, whatever).*
>>
>> To make it clear using code:
>>
>> func foo(s []T) {...}
>> // can this function be treated as following 3-parameter function?
>> func foo(p Pointer, length, capa int) {...}
>>
>> // Note: type is not the point here,
>> //   and we just need 3 parameters, while there maybe more
>>
>> You may wonder why I have this question? Here is where everything started:
>>
>> func foo(s []int) {
>>  s := []int{1,2,3}
>>  foo(s)
>>  fmt.Println(s)
>> }
>>
>> Everyone know it will print [1 2 3], not [1 2 3 100]. But what I am
>> interested in, is *whether the function call does modify the memory*
>> just after the tail of element 3, to an int of 100. If my assumption above
>> is right, then the modification may happen.
>>
>> So, I made some experiments.
>>
>> package main
>>
>> import "fmt"
>> import "strconv"
>>
>> type Node struct {
>> val int
>> left *Node
>> right *Node
>> }
>>
>> func (n Node) String () string {
>> return strconv.Itoa(n.val)
>> }
>>
>> func newNode(val int) *Node {
>> return {val, nil, nil}
>> }
>>
>> // Given a binary tree and a target sum, find all root-to-leaf paths
>> // where each path's sum equals the given sum
>> func bs(root *Node, path []*Node, now int, target int, ret *[][]*Node) {
>> if root == nil {
>> return
>> }
>> path = append(path, root)
>> now += root.val
>> if now == target && root.left == nil && root.right == nil {
>> *ret = append(*ret, path)
>> return
>> }
>> bs(root.left, path, now, target, ret)
>> bs(root.right, path, now, target, ret)
>> }
>>
>> func main() {
>> // a simple tree like:
>> //0
>> //   / \
>> //  1   2
>> root := newNode(0)
>> left := newNode(1)
>> right := newNode(2)
>> root.left = left
>> root.right = right
>>
>> ret := [][]*Node{}
>> bs(root, make([]*Node, 0, 10), 0, 1, )
>> fmt.Println(ret)
>> }
>>
>>
>> As the code above, it is a function to find all root-to-leaf paths where
>> each path's sum equals the given sum in a binary tree, and, I make a simple
>> test case which there are only 3 nodes: one root, two children, with values
>> of 0, 1 and 2.
>>
>> Say, I want to find the paths of sum of 1. So, I call this function as:
>>
>> bs(root, make([]*Node, 0, 10), 0, 1, )
>>
>> It is petty common to make a guess that, this call will give us a final
>> result of [0, 1], which is obviously the correct answer, however, it gives
>> us [0, 2], try yourself if you don't believe:
>>
>> https://play.golang.org/p/hSKIOaVK2S
>>
>> The algorithm here is correct, don't worry about it. Instead, please pay
>> attention to the second parameter of bs(). It makes a slice which has no
>> element in it, and has a capacity of 10. What if we change this parameter
>> to:
>>
>> bs(root, make([]*Node, 0, 1), 0, 1, )
>>
>> Yes, just make the slice's capacity as 1. This time you will get the
>> right answer, [0, 1]. Try yourself if you are interested.
>>
>> Here is my understanding of this strange problem, feel free to point out
>> anything wrong:
>>
>> Still the simplest code:
>>
>> package main
>>
>> import (
>> "fmt"
>> )
>>
>> func foo(s []int) {
>> fmt.Println(len(s), cap(s))   // 3 3
>> s = append(s, 100)
>> fmt.Println(len(s), cap(s))   // 4 8
>> }
>>
>> func main() {
>> s := []int{1,2,3}
>> foo(s)
>> fmt.Println(len(s), cap(s))   // 3 3
>> fmt.Println(s)// [1 2 3]
>> }
>>
>> When the function foo() called, it has 3 parameters: a pointer of the
>> first element of the array, which points to the element 1, an int for array
>> length: 3, an int for array capacity: 3.  In foo(), it tries to append 100
>> at the tail, but there is no room for it, according to the parameter of
>> capacity it receive, so, it 

[go-nuts] Re: Static race detector?

[gary . willoughby]

You might be interested in this:

Golang UK Conference 2016 - Nicholas Ng - Static Deadlock Detection for Go:
https://www.youtube.com/watch?v=NOQdkbp-950

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] On Concurrent-Safety

[Dave Cheney]

It depends who can see that variable, if this is a global variable, or its 
address has been taken it is not safe. If the variable is private the goroutine 
then it is safe becase there is no race on private variables. Remember a 
variable of type chan is just a pointer. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] On Concurrent-Safety

[dc0d]

Is setting a channel to nil, or to another channel, concurrent safe?

I guess it is not safe. Because it's a normal variable carrying the 
channel. But Go does some hidden optimizations, I wondered if this one is 
of those cases too.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Static race detector?

[Egon]

On Tuesday, 13 September 2016 09:15:57 UTC+3, Egon wrote:
>
> On Tuesday, 13 September 2016 08:53:39 UTC+3, josvazg wrote:
>>
>> Why does Go does not have a static race detector? What are the 
>> difficulties?
>> (just pointing me to the right docs to read is fine)
>>
>> I was thinking in a naive high level way...
>>  
>> If Rust can do it, using its ownership model, Go could have a tool 
>> checking the AST that analyzes variable scopes and checks globals, 
>> variables passed to goroutines and to/from channels.
>>
>> If one such followed variable is found in any piece of code to be 
>> accessed from more than 2 places and any access is a write, bam! flag that 
>> code with an error.
>>
>> What is this not possible or too difficult? too many corner cases? not 
>> enough supporting info in the code? (that could be build by the tool)
>> Or maybe is a work in progress,? maybe 100% reliable escape analysis is 
>> required beforehand?
>>
>
> One WIP http://www.doc.ic.ac.uk/~cn06/pub/2016/dingo/
>
> It's mainly -- *someone has to put in the effort*.
>
> AFAIR the main difficulty lies in buffered channels.
>

Oh, forgot to mention -- it's not a race detector, but rather a deadlock 
detector -- but, I guess, the principles are similar.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Static race detector?

[Egon]

On Tuesday, 13 September 2016 08:53:39 UTC+3, josvazg wrote:
>
> Why does Go does not have a static race detector? What are the 
> difficulties?
> (just pointing me to the right docs to read is fine)
>
> I was thinking in a naive high level way...
>  
> If Rust can do it, using its ownership model, Go could have a tool 
> checking the AST that analyzes variable scopes and checks globals, 
> variables passed to goroutines and to/from channels.
>
> If one such followed variable is found in any piece of code to be accessed 
> from more than 2 places and any access is a write, bam! flag that code with 
> an error.
>
> What is this not possible or too difficult? too many corner cases? not 
> enough supporting info in the code? (that could be build by the tool)
> Or maybe is a work in progress,? maybe 100% reliable escape analysis is 
> required beforehand?
>

One WIP http://www.doc.ic.ac.uk/~cn06/pub/2016/dingo/

It's mainly -- *someone has to put in the effort*.

AFAIR the main difficulty lies in buffered channels.

+ Egon

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Static race detector?

[Jan Mercl]

On Tue, Sep 13, 2016 at 7:53 AM josvazg  wrote:

> Why does Go does not have a static race detector? What are the
difficulties?

For example, the halting problem. Ok, now seriously.

> If Rust can do it, using its ownership model, Go could have a tool
checking the AST that analyzes variable scopes and checks globals,
variables passed to goroutines and to/from channels.
>
> If one such followed variable is found in any piece of code to be
accessed from more than 2 places and any access is a write, bam! flag that
code with an error.

I don't speak Rust, but AFAIK, its model of ownership does not provide
static race detecting, but can prove a race cannot happen, ie. no two paths
can concurrently read and write the same data. But a Go program can be
perfectly reasonable and completely race free even if it can be proven by
Rust's model as having a potential for races - assuming the Go program uses
synchronization properly. Proper synchronization avoids races, but it is
not possible in the general case to prove the synchronization is used
properly, because - the halting problem etc.

OTOH, a program proving that a particular Go program cannot have a race is
IMO possible to write. What I doubt is its value, because Go code which can
be proven race-free is probably only a small subset of Go code people write.

-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.