Re: CursorMark, batch size/speed

2019-06-12 Thread Erick Erickson
If there’s any chance of using Streaming for this rather than re-querying the data using CursorMark, it would solve a lot of these issues. > On Jun 12, 2019, at 3:26 PM, Mikhail Khludnev wrote: > > Every cursorMark request goes through full results. Previous results just > bypass scoring heap. S

Re: CursorMark, batch size/speed

2019-06-12 Thread Mikhail Khludnev
Every cursorMark request goes through full results. Previous results just bypass scoring heap. So, reducing number of such request should reasonably reduce wall-clock time exporting all results. On Wed, Jun 12, 2019 at 11:59 PM Markus Jelsma wrote: > Hello, > > One of our collections hates Curso

CursorMark, batch size/speed

2019-06-12 Thread Markus Jelsma
Hello, One of our collections hates CursorMark, it really does. When under very heavy load the nodes can occasionally consume GBs additional heap for no clear reason immediately after downloading the entire corpus. Although the additional heap consumption is a separate problem that i hope anyo