* Const V <thst...@gmail.com> [220508 17:20]:
> Using r.ReadLine repeatedly I was able to read the full line in memory. It
> is working very fast.
>
> for isPrefix && err == nil {
> line, isPrefix, err = r.ReadLine()
> ln = append(ln, line...)
> }
>
> if strings.Contains(s, "error") {
>
> finds the substring very fast
>
> Now is coming the last step - writing 100MB string to STDOUT
> w io.Writer
> w.Write([]byte(s))
>
> It is too big and freezes with large string.
I'm not sure why w.Write would freeze; is your process starting to swap
and it is not really frozen but just taking a long time? Is it being
killed by the kernel's out-of-memory monitor?
However, you shouldn't convert the []byte to string and then back to
[]byte again. You are doing two huge allocations unnecessarily. Also,
your code using ReadLine is doing essentially the same thing as
ReadBytes, and the documentation for ReadLine suggests using ReadBytes
instead.
Does https://go.dev/play/p/mSZ-Ft6C8pZ do what you want?
Note that if you are trying to be _really_ efficient, you should look at
the "Why Is Grep So Fast" article mentioned previously in this thread.
I would allocate a slice of large (1M? 10M?) byte slices, using them as
a ring buffer, adding one []byte to the ring at a time as needed. Then
implement Boyer-Moore or Rabin-Karp in a way that handles the boundaries
between []byte buffers properly. The total size of the ring buffer will
max out at the longest line, which is necessary anyway, and you will
minimize allocations and copying of data.
...Marvin
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/Ynp49ktp2CHKjucs%40basil.wdw.
