[
https://issues.apache.org/jira/browse/SOLR-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878556#comment-17878556
]
Andrzej Bialecki commented on SOLR-17430:
-----------------------------------------
Originally this design was an evolution of a single buffer-based older design,
where the "filler" and "writer" phases ran sequentially in the same thread. I
agree that something we initially thought would be a simple extension ended up
quite complicated :)
[~jbernste] and I ran several benchmarks using the old and the current design,
which showed big performance improvements in the current design. I think that
these speedups benefited from the bulk (buffer-based) operations for both read
and write sides of the process. Using a queue definitely simplifies the design
but I'm worried we may lose some of these performance gains when processing is
done item-by-item and not in bulk. OTOH this may not be such a huge factor
overall, and if it allows us to simplify the code and better control the flow,
then it may be worth it even with some performance penalty.
> Redesign ExportWriter / ExportBuffers to work better with large batchSizes
> and slow consumption
> -----------------------------------------------------------------------------------------------
>
> Key: SOLR-17430
> URL: https://issues.apache.org/jira/browse/SOLR-17430
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Reporter: Chris M. Hostetter
> Priority: Major
>
> As mentioned in SOLR-17416, the design of the {{ExportBuffers}} class used by
> the {{ExportHandler}} is brittle and the absolutely time limit on how long
> the buffer swapping threads will wait for eachother isn't suitable for very
> long running streaming expressions...
> {quote}The problem however is that this 600 second timeout may not be enough
> to account for really slow downstream consumption of the data. With really
> large collections, and really complicated streaming expressions, this can
> happen even when well behaved clients that are actively trying to consume
> data.
> {quote}
> ...but another sub-optimal aspect of this buffer swapping design is that the
> "writer" thread is initially completely blocked, and can't write out a single
> document, until the "filler" thread has read the full {{batchSize}} of
> documents into it's buffer and opted to swap. Likewise, after buffer
> swapping has occured at least once, any document in the {{outputBuffer}} that
> the writer has already processed hangs around, taking up ram, until the next
> swap, while one of the threads is idle. If {{{}batchSize=30000{}}}, and the
> "filler" thread is ready to go with a full {{fillBuffer}} while the "writer"
> has only been able to emit 29999 of the documents in it's {{outputBuffer}}
> documents before being blocked and forced to wait (due to the downstream
> consumer of the output bytes) before it can emit the last document in it's
> batch – that means both the "writer" thread and the "filler" thread are
> stalled, taking up 2x the batchSize of ram, even though half of that is data
> that is no longer needed.
> The bigger the {{batchSize}} the worse the initial delay (and steady state
> wasted RAM) is.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
