The solution you proposed is better now, but if the log produce speed is
still too large,
I think the printf function should not use a new goroutine( by go
func(){r:=... bufferchan<-r }), because the internal
bytes.Buffer{} might reaches the certain size always, it cannot provide the
ablibity to write so much log, the goroutine num might grow fastly.
On Sunday, March 17, 2019 at 1:29:23 PM UTC+8, Burak Serdar wrote:
>
> On Sat, Mar 16, 2019 at 11:10 PM <caij...@bytedance.com <javascript:>>
> wrote:
> >
> > I'm not sure if the log produce more than consume by the func 'centrale'
> in the same period of time, will 'buff' be larger and larger and cost too
> much memory.
> > For example, The log comes from network and write into a disk but
> network speed is greater than disk write speed. So is there any way to
> control the 'buff' length.
>
> There might be a simpler solution, but it needs to be tuned and tested
> to see if it behaves better: you can use a buffered string channel to
> store log messages. Your writer goroutine would read from this
> channel, and store the log messages in an internal bytes.Buffer{}
> until it reaches a certain size, and then writes it out. This should
> limit the memory usage based on the channel buffer size and buffer
> threshold. You should also have a time.Ticker to write the buffer
> periodically if log generation slows down.
>
>
> >
> > On Sunday, March 17, 2019 at 12:02:06 AM UTC+8, Thomas S wrote:
> >>
> >> Hello,
> >>
> >> I have a software needing a lot of logs when processing.
> >>
> >> I need to :
> >> 1- Improve the processing time by doing non-blocking logs
> >> 2- Be safe with goroutines parallel logs
> >>
> >> fmt package doesn't match with (1) & (2)
> >> log package doesn't match with (1)
> >>
> >> Here is my proposal. What do you think ?
> >>
> >> Design :
> >>
> >> [Log functions] -----channel---->[Buffering function (goroutine)]
> ----channel----> [Printing function (goroutine)]
> >>
> >>
> >> package glog
> >>
> >> import (
> >> "container/list"
> >> "encoding/json"
> >> "fmt"
> >> )
> >>
> >> /*
> >> ** Public
> >> */
> >>
> >> func Println(data ...interface{}) {
> >> bufferChan <- fmt.Sprintln(data...)
> >> }
> >>
> >> func Print(data ...interface{}) {
> >> bufferChan <- fmt.Sprint(data...)
> >> }
> >>
> >> func Printf(s string, data ...interface{}) {
> >> go func() {
> >> r := fmt.Sprintf(s, data...)
> >> bufferChan <- r
> >> }()
> >> }
> >>
> >> /*
> >> ** Private
> >> */
> >>
> >> var bufferChan chan string
> >> var outChan chan string
> >>
> >> func init() {
> >> bufferChan = make(chan string)
> >> outChan = make(chan string)
> >> go centrale()
> >> go buffout()
> >> }
> >>
> >> func centrale() {
> >> var buff *list.List
> >> buff = list.New()
> >> for {
> >> if buff.Len() > 0 {
> >> select {
> >> case outChan <- buff.Front().Value.(string):
> >> buff.Remove(buff.Front())
> >> case tmp := <-bufferChan:
> >> buff.PushBack(tmp)
> >> }
> >>
> >> } else {
> >> tmp := <-bufferChan
> >> buff.PushBack(tmp)
> >> }
> >> }
> >> }
> >>
> >> func buffout() {
> >> for {
> >> data := <-outChan
> >> fmt.Print(data)
> >> }
> >> }
> >>
> >>
> >>
> >> It works well for now, I want to be sure to not miss anything as it's a
> very important part of the code.
> >>
> >> Thank you for your review.
> >>
> > --
> > You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com <javascript:>.
> > For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
