* Const V <thst...@gmail.com> [220508 17:20]: > Using r.ReadLine repeatedly I was able to read the full line in memory. It > is working very fast. > > for isPrefix && err == nil { > line, isPrefix, err = r.ReadLine() > ln = append(ln, line...) > } > > if strings.Contains(s, "error") { > > finds the substring very fast > > Now is coming the last step - writing 100MB string to STDOUT > w io.Writer > w.Write([]byte(s)) > > It is too big and freezes with large string.
I'm not sure why w.Write would freeze; is your process starting to swap and it is not really frozen but just taking a long time? Is it being killed by the kernel's out-of-memory monitor? However, you shouldn't convert the []byte to string and then back to []byte again. You are doing two huge allocations unnecessarily. Also, your code using ReadLine is doing essentially the same thing as ReadBytes, and the documentation for ReadLine suggests using ReadBytes instead. Does https://go.dev/play/p/mSZ-Ft6C8pZ do what you want? Note that if you are trying to be _really_ efficient, you should look at the "Why Is Grep So Fast" article mentioned previously in this thread. I would allocate a slice of large (1M? 10M?) byte slices, using them as a ring buffer, adding one []byte to the ring at a time as needed. Then implement Boyer-Moore or Rabin-Karp in a way that handles the boundaries between []byte buffers properly. The total size of the ring buffer will max out at the longest line, which is necessary anyway, and you will minimize allocations and copying of data. ...Marvin -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/Ynp49ktp2CHKjucs%40basil.wdw.