Hi, thanks for help!
But the problem might just be that you're writing many more bytes to > your encoder than you're reading from the file. > Did you mean i should write it so ? Write(srcBuf[0:lenRead]) nrRead, err := src.Read(bufRead); // read from file if err!= nil { break } fmt.Printf("read read len: %d\n", nrRead) nwEnc, err := encoder.Write(bufRead[0:nrRead]) // write to encoder This means the size of buf2 could be much larger than the original > file and also full of junk data from previous reads and zeroed bytes. > One q i have here, is not buf2 supposed to be drained by the following call to Read, at least partially _, err = buf2.Write(r) // write to decoder fmt.Println(err) if decoder==nil { decoder = flate.NewReader(buf2) } r2 := make([]byte, dReadSize*2) _, err = decoder.Read(r2) // read from decoder // <- here, it should remove some data of buf2, no ? fmt.Println(err) Besides that, i totally agree this example code is full of wrongness, I rewrote it, package main import ( "fmt" "os" "bytes" // "encoding/base64" "compress/flate" "io" "time" ) func main () { // okSize := 16384 // koSize := 64384 // koSize := 16385 // yes 1 more and it breaks :s // change read size to 16384 and everything is ok fReadSize := 16385 dReadSize := 64384 // dread seems not impacting src, _ := os.Open("src.avi") bufRead := make([]byte, fReadSize) bufEnc := new(bytes.Buffer) encoder, _ := flate.NewWriter(bufEnc, 9) bufDec := new(bytes.Buffer) var decoder io.ReadCloser for { fmt.Println("-----------") nrRead, err := src.Read(bufRead); // read from file if err!= nil { break } fmt.Printf("read read len: %d\n", nrRead) nwEnc, err := encoder.Write(bufRead[0:nrRead]) // write to encoder fmt.Println(err) fmt.Printf("encode write len: %d\n", nwEnc) err = encoder.Flush() fmt.Println(err) encSlice := bufEnc.Bytes() // read from encoder // fmt.Println(encSlice[0:20]) fmt.Printf("encode read len: %d\n", len(encSlice)) fmt.Printf("bufEnc len: %d\n", bufEnc.Len()) fmt.Printf("bufEnc cap: %d\n", bufEnc.Cap()) bufEnc.Truncate(0) // fmt.Println("_______") // fmt.Println(encSlice[0:20]) // fmt.Println(r) nwDec, err := bufDec.Write(encSlice) // write to decoder fmt.Println(err) fmt.Printf("decode write len: %d\n", nwDec) if decoder==nil { decoder = flate.NewReader(bufDec) } sliceDec := make([]byte, dReadSize*2) nrDec, err := decoder.Read(sliceDec) // read from decoder fmt.Println(err) fmt.Printf("decode read len: %d\n", nrDec) fmt.Printf("bufDec len: %d\n", bufDec.Len()) fmt.Printf("bufDec cap: %d\n", bufDec.Cap()) time.Sleep( 1 * time.Second) } } Which after few iterations will yield read read len: 16385 <nil> encode write len: 16385 <nil> encode read len: 15150 bufEnc len: 15150 bufEnc cap: 16320 <nil> decode write len: 15150 <nil> decode read len: 8 bufDec len: 51234 bufDec cap: 140569 So definitely the decoder is holding on something. Its cap and len are irremediably growing. The thing now is if I try to read it twice time consecutively, the second will read will always yield 0 bytes and provide me an UnexpectedEOF error. Does this error leaves the decoder in a unrecoverable state ? I m unsure. Looks likes i can t drain it :x Le lundi 4 juillet 2016 06:17:51 UTC+2, Jesse McNelis a écrit : > > On Mon, Jul 4, 2016 at 3:35 AM, mhhcbon <cpasmabo...@gmail.com > <javascript:>> wrote: > > Hi, > > > > I have this program which reads file, flate encode then flate decode the > > data. > > > > I noticed that when i used different size for the slice of []byte to > read > > data, the program will retain memory when the size is > 16384. > > When its lower than this value everything is fine, but 16385 breaks. > > > > I don t quite understand the reason of this behavior, can someone help > me to > > understand what s going on there ? > > I can't see anywhere that this program could be holding on to extra > memory it's not using. > But the problem might just be that you're writing many more bytes to > your encoder than you're reading from the file. > > _, err := src.Read(b); // read from file > > Read() isn't required to fill the whole buffer it's given. It can read > a single byte and return. > Because you're ignoring the value telling you how many bytes it read > you're passing the whole 16385 slice to your encoder even though you > might have read much less than 16385 bytes. > > This means the size of buf2 could be much larger than the original > file and also full of junk data from previous reads and zeroed bytes. > > Read() is a low level call that you should avoid calling directly > because it's tricky to get right. > > For an example of how to properly call a Read() see the implementation > of io.Copy() > https://golang.org/src/io/io.go?#L366 > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.