[
https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891085#action_12891085
]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
This is looking awesome Michael! I love the removal of *PerThread --
they are all logically absorbed into DWPT, so everything is now per
thread.
I still see usage of docStoreOffset, but aren't we doing away with
shared doc stores with the cutover to DWPT?
I think you can further simplify DocumentsWriterPerThread.DocWriter;
in fact I think you can remove it & all subclasses in consumers! The
consumers can simply directly write their files. The only reason this
class was created was because we have to interleave docs when writing
the doc stores; this is no longer needed since doc stores are again
private to the segment. I think we don't need PerDocBuffer, either.
And this also simplifies RAM usage tracking!
Also, we don't need separate closeDocStore; it should just be closed
during flush.
I like the ThreadAffinityDocumentsWriterThreadPool; it's the default
right (I see some tests explicitly setting in on IWC; not sure why)?
We should make the in-RAM deletes impl somehow pluggable?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: Realtime Branch
>
> Attachments: lucene-2324.patch, lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
