Re: [go-nuts] Channels: selecting the last element
Just realized what you meant about buffered channels. I'm wrong, and sorry
for the confusion.
+1 to your original solution (combined with setting channels to nil).
Diane Looney
On Tue, Dec 18, 2018 at 11:31 AM Chris Burkert
wrote:
> Hello Ian, all,
> yes, the workers generate multiple results. I was able to use your
> proposal with the waiting goroutine which closes the channel. Unfortunately
> my initial minimal example was not so minimal. It is a little more
> complicated, as I have multiple "result" channels with different types and
> I don't know how many values I get. So I can't just range over one channel
> but instead use select with the ok from "val, ok := <-channel" to count the
> closed channels which finally ends the loop. Below is what I came up with
> (using Johns approach to run it multiple times) and it seems to be
> deterministic.
> Many thanks!
> PS: with buffered channels I have to disable closed channels with "channel
> = nil" to make it work.
>
> package main
>
> import (
> "fmt"
> "sync"
> )
>
> func do(fc chan<- int, rc chan<- string) {
> fc <- 42
> fc <- 43
> rc <- "foo"
> }
>
> func main() {
> w := 4
> r := 10
> n := 0
> for i := 0; i < r; i++ {
> n = n + run(w)
> }
> fmt.Printf("Got %d, expected %d\n", n, 3*w*r)
> }
>
> func run(worker int) int {
> fc := make(chan int)
> rc := make(chan string)
> var wg sync.WaitGroup
> wg.Add(worker)
> for i := 0; i < worker; i++ {
> go func() {
> defer wg.Done()
> do(fc, rc)
> }()
> }
> go func() {
> wg.Wait()
> close(fc)
> close(rc)
> }()
> n := 0
> closed := 0
> for closed < 2 {
> select {
> case _, ok := <-fc:
> if ok {
> n++
> } else {
> closed++
> }
> case _, ok := <-rc:
> if ok {
> n++
> } else {
> closed++
> }
> }
> }
> return n
> }
>
> Am Di., 18. Dez. 2018 um 15:50 Uhr schrieb Ian Lance Taylor <
> i...@golang.org>:
>
>> On Tue, Dec 18, 2018 at 5:35 AM Chris Burkert
>> wrote:
>> >
>> > I have a couple of goroutines sending multiple results over a channel -
>> a simple fan-in. They signal the completion on a done channel. Main selects
>> on the results and done channel in parallel. As the select is random main
>> sometimes misses to select the last result. What would be the idiomatic way
>> to prevent this and completely drain the result channel?
>> >
>> > Here is a minmal example which sometimes prints one 0 but should always
>> print two of them:
>> >
>> > package main
>> >
>> > import (
>> > "fmt"
>> > )
>> >
>> > func do(rc chan<- int, dc chan<- bool) {
>> > rc <- 0
>> > dc <- true
>> > }
>> >
>> > func main() {
>> > worker := 2
>> > rc := make(chan int, worker)
>> > done := 0
>> > dc := make(chan bool, worker)
>> > for i := 0; i < worker; i++ {
>> > go do(rc, dc)
>> > }
>> > for done < worker {
>> > select {
>> > case <-dc:
>> > done++
>> > case r := <-rc:
>> > fmt.Println(r)
>> > }
>> > }
>> > }
>>
>>
>> I assume the workers can generate multiple results, as otherwise the
>> done marker seems pointless. In general the simplest way to signal
>> completion on a channel is to call close. The simplest way to call
>> close on a fan-in is to have another goroutine that waits for the
>> other goroutines and closes the channel. That might look like
>>
>> package main
>>
>> import (
>> "fmt"
>> "sync"
>> )
>&g
Re: [go-nuts] Channels: selecting the last element
I'm not sure this code behaves as you expect it to. I don't think there's
any magic in place to prevent select from picking the same closed channel
more than once. For example:
https://play.golang.org/p/nVvsBxww_Rb
package main
import (
"fmt"
"sync"
)
func test() {
c1 := make(chan bool, 1000)
c2 := make(chan bool)
closed := 0
closedByC1 := 0
closedByC2 := 0
n := 0
for i := 0; i < 1000; i++ {
c1 <- true
}
close(c1)
close(c2)
for closed < 2 {
select {
case _, ok := <-c1:
if !ok {
closed++
closedByC1++
} else {
n++
}
case _, ok := <-c2:
if !ok {
closed++
closedByC2++
} else {
n++
}
}
}
fmt.Printf("%v\t%v\t%v\t%v\n", n, closed, closedByC1, closedByC2)
}
func main() {
wg := sync.WaitGroup{}
wg.Add(20)
for i := 0; i < 20; i++ {
go func() {
defer wg.Done()
test()
}()
}
wg.Wait()
}
I think if you want to do this, you should instead go down the route of
multiple goroutines that all signal on a waitgroup for each job type.
https://play.golang.org/p/DG9Aes_rqoR
package main
import (
"fmt"
"sync"
)
func do(fc chan<- int, rc chan<- string) {
fc <- 42
fc <- 43
rc <- "foo"
}
func main() {
w := 4
r := 10
n := 0
for i := 0; i < r; i++ {
n = n + run(w)
}
fmt.Printf("Got %d, expected %d\n", n, 3*w*r)
}
func run(worker int) int {
fc := make(chan int, 10)
rc := make(chan string)
wg1 := sync.WaitGroup{}
wg1.Add(worker)
for i := 0; i < worker; i++ {
go func () {
defer wg1.Done()
do(fc, rc)
}()
}
wg2 := sync.WaitGroup{}
wg2.Add(2)
n:=0
go func() {
defer wg2.Done()
for {
_, ok := <- fc
if ! ok {
return
}
n++
}
}()
go func() {
defer wg2.Done()
for {
_, ok := <- rc
if ! ok {
return
}
n++
}
}()
wg1.Wait()
close(fc)
close(rc)
wg2.Wait()
return n
}
Diane Looney
On Tue, Dec 18, 2018 at 11:31 AM Chris Burkert
wrote:
> Hello Ian, all,
> yes, the workers generate multiple results. I was able to use your
> proposal with the waiting goroutine which closes the channel. Unfortunately
> my initial minimal example was not so minimal. It is a little more
> complicated, as I have multiple "result" channels with different types and
> I don't know how many values I get. So I can't just range over one channel
> but instead use select with the ok from "val, ok := <-channel" to count the
> closed channels which finally ends the loop. Below is what I came up with
> (using Johns approach to run it multiple times) and it seems to be
> deterministic.
> Many thanks!
> PS: with buffered channels I have to disable closed channels with "channel
> = nil" to make it work.
>
> package main
>
> import (
> "fmt"
> "sync"
> )
>
> func do(fc chan<- int, rc chan<- string) {
> fc <- 42
> fc <- 43
> rc <- "foo"
> }
>
> func main() {
> w := 4
> r := 10
> n := 0
> for i := 0; i < r; i++ {
> n = n + run(w)
> }
> fmt.Printf("Got %d, expected %d\n", n, 3*w*r)
> }
>
> func run(worker int) int {
> fc := make(chan int)
> rc := make(chan string)
> var wg sync.WaitGroup
> wg.Add(worker)
> for i := 0; i < worker; i++ {
> go func() {
> defer wg.Done()
> do(fc, rc)
> }()
> }
> go func() {
> wg.Wait()
> close(fc)
> close(rc)
> }()
> n := 0
> closed := 0
> for closed < 2 {
> select {
> case _, ok := <-fc:
> if ok {
> n++
> } else {
> closed++
> }
> case _, ok := <-rc:
> if ok {
>
Re: [go-nuts] JSON dynamic keys
In addition to Burak's suggestion, keep in mind that if you implement
MarshalJSON and/or UnmarshalJSON for certain structs, then those will be
used instead of the built-in functions. If you end up cleaning up your
struct definitions and just cant find the right way to annotate your fields
to get the behavior you want, then that might be your best bet.
example is in godoc at https://golang.org/pkg/encoding/json/ under
CustomMarshalJSON
Diane Looney
On Mon, Dec 3, 2018 at 10:53 AM Burak Serdar wrote:
> On Mon, Dec 3, 2018 at 8:27 AM Jeffrey Smith
> wrote:
> >
> > I'm trying to set mapping in elastic search in go and want to generate
> something like this.
> >
> > { "mappings": { "_doc": { "properties": { "title": { "type": "text",
> "store": true }, "date": { "type": "date", "store": true }, "content": {
> "type": "text" } } } } }
> >
>
> Is this close to what you're trying to do:
>
> https://play.golang.org/p/wYOwf6wUai7
>
> >
> > The _doc, title,date and content are all keys that will change depending
> on what mapping I am trying to generate.
> >
> > I have a basic struct layout but cant work out how to generate the
> proper JSON.
> >
> > type mappingData struct {
> > Mappings struct {
> > Doc struct {
> > Properties struct {
> > Elements []Fields
> > } `json:"properties"`
> > } `json:"_doc"`
> > } `json:"mappings"`
> > }
> >
> >
> > type Fields struct {
> > Type string `json:"type"`
> > Store bool `json:"store"`
> > }
> >
> > _doc will be from a string and I have a vector of structs that has
> different fields in.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts+unsubscr...@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
