I ended up creating an issue on this: https://github.com/golang/go/issues/44026
On Sat, Jan 30, 2021 at 9:12 PM Christian Worm Mortensen <c...@epust.dk> wrote: > Hi Mike, > > Thank you for your consideration. I think you exactly got the essence of > my question: How do I wait on all go routines to finish (or be blocked on > one or more channels) before advancing time. > > A key thing I would like from such a solution is that it does not require > too heavy modifications to the code to be tested or put restrictions on how > it can do things. > > I think it may be possible to solve it with some explicit check in / check > out as I think you also suggest. I guess in essence you will check out > before you call select and check in again after select is done waiting. I > think this will still not work if buffered channels are used. But maybe if > buffered channels are mocked, it may be doable. > > I think I may want to make a feature request on this. I see several > options: > > * Make a version of runtime.gosched that only returns when no other go > routines can run > * Make it possible to read the number of go routines that are ready to > run. You could then make a loop where you call runtime.gosched until that > value is 0. > * Make it possible to start a special go routine when the system is > deadlocked. > > One problem is what to do if the program is waiting on external IO such as > the completion of an HTTP request. I guess in an ideal solution it would be > possible for the program to decide if it will advance time in that > situation or not. > > Please let me know if you have any ideas of other things to put into the > feature request. > > Thanks, > > Christian > > On Fri, Jan 29, 2021 at 9:25 PM mspr...@us.ibm.com <mspre...@us.ibm.com> > wrote: > >> Volker: injecting sleep is a nice idea, in the general vein that Jesper >> said of injecting time. However, as soon as we zoom out a step and need to >> test both that generator and the goroutine(s) consuming and acting upon >> that channel activity, we get back to the essence of the original question: >> how to test when we have a bunch of goroutines doing stuff and the test >> needs to wait for them all to finish before advancing time? >> >> FYI, in Kubernetes we have done something similar to the Facebook clock >> package --- but recently we have called out the narrower interface used by >> code that only reads time. See PassiveClock in >> https://github.com/kubernetes/utils/blob/master/clock/clock.go and >> https://github.com/kubernetes/apimachinery/blob/master/pkg/util/clock/clock.go >> (yeah, we have two forked lines of development of this clock thing, sigh). >> >> The pattern of using channel activity to coordinate asynchronous activity >> is inherently inimical to what the original poster asked for. An >> alternative is to define clocks that run procedures rather than do channel >> sends. See the EventClock in >> https://github.com/kubernetes/apiserver/blob/master/pkg/util/flowcontrol/fairqueuing/testing/clock/event_clock.go >> . A mocked one of those could know when all the timed activities have >> completed --- if all the timed activities were synchronously contained in >> EventFuncs. Sadly this is too restrictive a pattern for a lot of real >> code. You will see in that package an additional idea: explicitly tracking >> (at "user level") when the goroutines in question block/unblock. This is >> painful, but I see no better way (given the golang runtime interface as it >> is defined today). >> >> Regards, >> Mike >> >> On Friday, January 29, 2021 at 10:11:34 AM UTC-5 Volker Dobler wrote: >> >>> One way to do this is have an internal implementation like >>> func generatorImpl(sleep func(time.Duration)) <-chan int >>> and func generator just calls that one with time.Sleep. >>> Tests are done against generatorImpl where you know have >>> detailed control of how much (typically none) time is >>> actually slept. >>> >>> Expiration of cookies is tested in that way, see e.g. >>> https://golang.org/src/net/http/cookiejar/jar.go#L159 >>> So while technically Jar.Cookies is never tested the >>> risk is basically nil. >>> >>> V. >>> On Thursday, 28 January 2021 at 22:15:50 UTC+1 Christian Worm Mortensen >>> wrote: >>> >>>> Hi! >>>> >>>> Suppose I want to unit test this function: >>>> >>>> func generator() <-chan int { >>>> ret := make(chan int) >>>> go func() { >>>> for i := 0; i < 10; i++ { >>>> ret <- i >>>> time.Sleep(time.Second) >>>> } >>>> }() >>>> return ret >>>> } >>>> >>>> What is a good way to do that? One way is to do it is like this: >>>> >>>> func testGenerator() { >>>> start := time.Now() >>>> g := generator() >>>> for i := 0; i < 10; i++ { >>>> v := <-g >>>> if v != i { >>>> panic("Wrong value") >>>> } >>>> } >>>> elapsed := time.Now().Sub(start) >>>> if elapsed < 9*time.Second || elapsed > 11*time.Second { >>>> panic("Wrong execution time") >>>> } >>>> } >>>> >>>> However there are several issues with this: >>>> >>>> 1) The unit test takes a long time to run - 10 seconds. >>>> 2) The unit test is fragile to fluctuations in CPU availability >>>> 3) The unit test is not very accurate >>>> >>>> Of course this is a simple example. But what if I want to test a >>>> complicated piece of code with many go routines interacting in complicated >>>> ways and with long timeouts? >>>> >>>> In other programming languages, I have been able to implement a form of >>>> virtual time which increases only when all threads are waiting for time to >>>> increase. This allows functions like generator above to be tested basically >>>> instantly and this has been extremely useful for me in many projects over >>>> the years. >>>> >>>> Can I do something similar in Go? I would expect I would need to wrap >>>> time.Now, time.Sleep and time.After which I will be happy to do. >>>> >>>> I can see that Go has a deadlock detector. If somehow it was possible >>>> to have Go start a new Go routine when a deadlock was detected, I think it >>>> would be pretty straight forward to implement virtual time as described. I >>>> could then do something like: >>>> >>>> runtime.registerDeadlockCallback(func () { >>>> // Increase virtual time and by that: >>>> // * Make one or more wrapped time.Sleep calls return or >>>> // * Write to one or more channels returned by wrapped time.After. >>>> }) >>>> >>>> Obviously this would only be needed for test code, not production code. >>>> >>>> Thanks, >>>> >>>> Christian >>>> >>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "golang-nuts" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/golang-nuts/Y9Ccen0uMcs/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> golang-nuts+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/golang-nuts/9f5fa53a-64ca-483a-8e63-bae5c061e569n%40googlegroups.com >> <https://groups.google.com/d/msgid/golang-nuts/9f5fa53a-64ca-483a-8e63-bae5c061e569n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CABTkUoa%2Bt5QWXzNmC_du7DzRswczLGiURq_5iOYGywxMqC%3Ddxg%40mail.gmail.com.
