[go-nuts] Re: Go channels overused and hyped?

[Caleb Doxsey]

Channels are very useful and necessary to really get your program using all 
the resources at your disposal. Doing that with locks or callbacks is error 
prone and makes solving some problems all but impossible. Let me give you 
an example of a pattern we used a few days ago:

We were processing message from a queue, writing some data to Cassandra, 
and then committing offsets when the work was completed. It was really 
important for the commits to come out in the same order they came in, so 
that we never commit an offset when we haven't actually written all the 
data. (on crash or restart its ok to process the same messages again 
because the writing is idempotent)

Modeled as a pipeline we had a few stages:

intake -> decode -> encode -> write to cassandra -> commit

The naive approach would be to spin up a bunch of writer goroutines, but 
then the order of messages to commit would be unpredictable. Instead we 
created two slices of channels:

ins := make([]chan intakeMessage, options.workers)
outs := make([]chan commitMessage, options.workers)

And spun up goroutines that read from one of the input channels and wrote 
to the corresponding out channel:

for i := 0; i < options.workers; i++ {
ins[i] = make(chan intakeMessage, 1)
outs[i] := make(chan commitMessage, 1)
go worker(ins[i], outs[i])
}

We then use these slices as a circular buffer and keep track of two 
pointers, one for the next available in and another for the next remaining 
out. There are 3 cases:

   1. The slice is empty, in which case we insert at 0 and increment the in 
   counter
   2. The slice is full (all workers are busy), in which case we wait until 
   a result is pushed into the channel at the out counter
   3. We're somewhere in between, in which case we use a select on either 
   the next available in channel or the next remaining out counter

So all the commits come out in the order they came in, we get nice 
parallelism and there's no blocking or polling.

There is overhead to channels but if you send messages of adequate size 
you'll barely notice it in real programs. Batch and send slices.

The thing I find remarkable about channels and goroutines is how often the 
solutions for improving the performance of a Go program are almost exactly 
the same as the solutions for distributing that program across multiple 
machines. The patterns really map well to these kinds of problems.

On Tuesday, August 8, 2017 at 2:01:12 AM UTC-4, snmed wrote:
>
> Hi Gophers
>
> I stumbled over a nice and very interesting Blog entry "Go channels are 
> bad and you should feel bad 
> "
>  
> , I would like to hear some opinions about that article
> from seasoned go developers. Because I just used go for a couple of months 
> for my private web projects and rarely get in touch with channels.
>
> By the way, that article is not a rant and the author likes go very much 
> as far as I can conclude from the article.
>
> Cheers snmed
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go channels overused and hyped?

[Jason Stillwell]

Here's the thing about channels in go. They're a primitive. Just like int 
is a primitive and string is a primitive. They're not an entire library, or 
framework. they're just a primitive.

In other languages, currency "tools" are usually provided as some part of a 
fully featured library, or package, or framework. But not in go. Go 
provides the primitive, and its up to you to use that to implement your 
actual concurrency tools.

You might try to store a date in an int. But there are lots of caveats 
along with doing so. Instead you take primitives like ints and structs and 
put them together to make a package and set of structures for dealing with 
Dates. You don't complain that int alone isn't good enough for doing all 
your date processing.

But a lot of devs see the channel and assume its some high level 
abstraction built with an API to deal with all your common situations, 
because thats what they are used to expecting from their other language 
experience. Its not though. Its just a building block. Build something with 
it, and then use that something.

On Monday, August 7, 2017 at 11:01:12 PM UTC-7, snmed wrote:
>
> Hi Gophers
>
> I stumbled over a nice and very interesting Blog entry "Go channels are 
> bad and you should feel bad 
> "
>  
> , I would like to hear some opinions about that article
> from seasoned go developers. Because I just used go for a couple of months 
> for my private web projects and rarely get in touch with channels.
>
> By the way, that article is not a rant and the author likes go very much 
> as far as I can conclude from the article.
>
> Cheers snmed
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go channels overused and hyped?

[Dave Cheney]

Channels are always going to be more expensive than using a lock if all your 
doing is timing the cost of the lock vs a channel; aka hello world of channels. 
Channels are there to send values between goroutines representing work to do, 
in which case that work has to be significantly more than the cost of the 
transmission otherwise it's easier to just do the work yourself. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go channels overused and hyped?

[Haddock]

CSP style concurrency in Go is the killer feature in Go. It makes 
concurrent programming so much easier. To understand this you need to have 
done some years of concurrent programming using asynchronous calls, 
threads, locks, semaphores and all that stuff. I have done that and I can 
say that CSP style programming in Go with the use of channels makes 
concurrent programming much much easier. Much easier to get things right 
from the beginning, much easier to fix things. I have actually spent years 
fixing other people's and my mistakes causing race conditions and 
deadlocks. People who have not experienced this have no considerable 
experience in concurrent programming and don't know what they are talking 
about. 

You have to mention that green threads (aka goroutines) are not preemptive, 
though. This is what you have to pay for what you get for the ease of 
concurrent programming using CSP. For many applications this is a very good 
trade-off. All the server-side applications written in Go confirm this.


Am Dienstag, 8. August 2017 08:01:12 UTC+2 schrieb snmed:
>
> Hi Gophers
>
> I stumbled over a nice and very interesting Blog entry "Go channels are 
> bad and you should feel bad 
> "
>  
> , I would like to hear some opinions about that article
> from seasoned go developers. Because I just used go for a couple of months 
> for my private web projects and rarely get in touch with channels.
>
> By the way, that article is not a rant and the author likes go very much 
> as far as I can conclude from the article.
>
> Cheers snmed
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Go channels overused and hyped?

[Konstantin Khomoutov]

On Tue, Aug 08, 2017 at 03:39:42AM -0700, snmed wrote:

> > There are trade-offs.
> >
> > Channels are easy to use for simple things, but complicated for complected 
> > things.
> >
> > Locking data-structures can easily introduce data-races (see The Little 
> > Book of Semaphores http://greenteapress.com/wp/semaphores/).
[...]
> Thank you for your reply, I myself used channel as a semaphore, i'm not 
> sure if that is a appropriate use case for channels.
[...]

Channels are quite good to implement simple counting semaphores (those
which have more than a single token / permit to provide): the length of
the channel used in this way is the quantity of permits the semaphore
has, a send to such channel is acquisition of a permit, and receiving
from it returning of the acquired permit back to the semaphore.

Acquisition hence naturally blocks if the semaphore has no free permits
and blocks the requesting goroutine until a free permit becomes
available.

You can easily combine this with a timer to get a counting semaphore
with a timeout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go channels overused and hyped?

[snmed]

Hi Dave

Thank you for your comment, I will read your talk on my way back home later 
and I'm sure i get some better insight in go concurrency.
But his argument about better performance of mutex over channels is still 
valid, maybe you have covered that in your talk, i will see later.

Cheers snmed


Am Dienstag, 8. August 2017 09:51:08 UTC+2 schrieb Dave Cheney:
>
> Everyone overused channels and goroutines at first. Why wouldn't you? 
> That's why you probably decided to try Go in the first place.
>
> I have a talk in Singapore a few months ago trying to explore this idea. 
> It sort of went in a different direction, but the conclusion might be 
> interesting for you.
>
> https://dave.cheney.net/paste/concurrency-made-easy.pdf
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go channels overused and hyped?

[snmed]

Hi Egon

Thank you for your reply, I myself used channel as a semaphore, i'm not 
sure if that is a appropriate use case for channels.

Anyhow your opionion is very welcome
 

Am Dienstag, 8. August 2017 09:26:35 UTC+2 schrieb Egon:
>
> There are trade-offs.
>
> Channels are easy to use for simple things, but complicated for complected 
> things.
>
> Locking data-structures can easily introduce data-races (see The Little 
> Book of Semaphores http://greenteapress.com/wp/semaphores/).
>
> The Game/Player example looks weird to me; there's one piece missing -- 
> without it, the full complexity is not seen.
>
> With regards to callbacks (context.Done) specifically, in one case you 
> control the goroutine it gets executed in, in the other not.
>
> Not closing the waiting channel inside context leaking goroutine is 
> moot... when you forget to invoke the callbacks, you will leak as well when 
> you have resource releasing in the callback.
>
> Channels are quite good for producer-consumer things, but tend to get very 
> complicated for dispatches.
>
> *tl;dr; channels and locking have trade-offs, instead of hopping on the 
> bandwagon, understand the trade-offs and pick the one that is most suitable 
> in a certain situation.*
>
> + Egon
>
> On Tuesday, 8 August 2017 09:01:12 UTC+3, snmed wrote:
>>
>> Hi Gophers
>>
>> I stumbled over a nice and very interesting Blog entry "Go channels are 
>> bad and you should feel bad 
>> "
>>  
>> , I would like to hear some opinions about that article
>> from seasoned go developers. Because I just used go for a couple of 
>> months for my private web projects and rarely get in touch with channels.
>>
>> By the way, that article is not a rant and the author likes go very much 
>> as far as I can conclude from the article.
>>
>> Cheers snmed
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go channels overused and hyped?

[as . utf8]

Author did a range over channel without select, making cancellation 
impossible without closing the channel or the process. However, author 
challenges user to solve cancellation problems with no selection, even 
saying he awaits for the user to accomplish this. So author will stop 
waiting when notified, or call them back, to say it's been done. Because it 
is not possible, we presume author is still blocked waiting on his 
completion signal, or worse, polling to see any completed the impossible.

In practice, the callback needs to live somewhere in memory. The select 
statement not so much; it could have provided cancellation by broadcasting 
through a "done" channel. For one reason or another, the author didn't do 
this and ranged through the channel instead.

On Monday, August 7, 2017 at 11:01:12 PM UTC-7, snmed wrote:
>
> Hi Gophers
>
> I stumbled over a nice and very interesting Blog entry "Go channels are 
> bad and you should feel bad 
> "
>  
> , I would like to hear some opinions about that article
> from seasoned go developers. Because I just used go for a couple of months 
> for my private web projects and rarely get in touch with channels.
>
> By the way, that article is not a rant and the author likes go very much 
> as far as I can conclude from the article.
>
> Cheers snmed
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go channels overused and hyped?

[Egon]

There are trade-offs.

Channels are easy to use for simple things, but complicated for complected 
things.

Locking data-structures can easily introduce data-races (see The Little 
Book of Semaphores http://greenteapress.com/wp/semaphores/).

The Game/Player example looks weird to me; there's one piece missing -- 
without it, the full complexity is not seen.

With regards to callbacks (context.Done) specifically, in one case you 
control the goroutine it gets executed in, in the other not.

Not closing the waiting channel inside context leaking goroutine is moot... 
when you forget to invoke the callbacks, you will leak as well when you 
have resource releasing in the callback.

Channels are quite good for producer-consumer things, but tend to get very 
complicated for dispatches.

*tl;dr; channels and locking have trade-offs, instead of hopping on the 
bandwagon, understand the trade-offs and pick the one that is most suitable 
in a certain situation.*

+ Egon

On Tuesday, 8 August 2017 09:01:12 UTC+3, snmed wrote:
>
> Hi Gophers
>
> I stumbled over a nice and very interesting Blog entry "Go channels are 
> bad and you should feel bad 
> "
>  
> , I would like to hear some opinions about that article
> from seasoned go developers. Because I just used go for a couple of months 
> for my private web projects and rarely get in touch with channels.
>
> By the way, that article is not a rant and the author likes go very much 
> as far as I can conclude from the article.
>
> Cheers snmed
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.