[
https://issues.apache.org/jira/browse/LUCENE-8438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16572977#comment-16572977
]
Dawid Weiss commented on LUCENE-8438:
-------------------------------------
So here's what I think should work with regard to concerns about stability, API
continuity and experimental implementation of this new ram-based Directory:
1) add those ByteBuffers* classes immediately to 7.x and 8.x line, together
with tests, mark them as experimental (do not change anything in existing
code). This allows people (like me) to play with those classes, provide
feedback and perhaps discover bugs.
2) mark RAMDirectory deprecated in 7.x and 8.x, schedule it for removal in a
later major version.
3) postpone all codec changes and other replacements until after 8.0 is
released, schedule them for 8.1, for example. I honestly don't think anything
can go bad in there, but it's software, after all. Can be proven correct, but
nobody knows until you run it. ;)
4) separate another subtask to replace all RAMDirectory uses in tests and other
places. These can also go to 8.1 after 8.0 branch is cut.
The above will require splitting this patch into at least two, but it's quite
trivial.
> RAMDirectory speed improvements and cleanup
> -------------------------------------------
>
> Key: LUCENE-8438
> URL: https://issues.apache.org/jira/browse/LUCENE-8438
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Priority: Minor
> Attachments: capture-1.png, capture-4.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> RAMDirectory screams for a cleanup. It is used and abused in many places and
> even if we discourage its use in favor of native (mmapped) buffers, there
> seem to be benefits of keeping RAMDirectory available (quick throw-away
> indexes without the need to setup external tmpfs, for example).
> Currently RAMDirectory performs very poorly under concurrent loads. The
> implementation is also open for all sorts of abuses – the streams can be
> reset and are used all around the place as temporary buffers, even without
> the presence of RAMDirectory itself. This complicates the implementation and
> is pretty confusing.
> An example of how dramatically slow RAMDirectory is under concurrent load,
> consider this PoC pseudo-benchmark. It creates a single monolithic segment
> with 500K very short documents (single field, with norms). The index is ~60MB
> once created. We then run semi-complex Boolean queries on top of that index
> from N concurrent threads. The attached capture-4 shows the result (queries
> per second over 5-second spans) for a varying number of concurrent threads on
> an AWS machine with 32 CPUs available (of which it seems 16 seem to be real,
> 16 hyper-threaded). That red line at the bottom (which drops compared to a
> single-threaded performance) is the current RAMDirectory. RAMDirectory2 is an
> alternative implementation I wrote that uses ByteBuffers. Yes, it's slower
> than the native mmapped implementation, but a *lot* faster then the current
> RAMDirectory (and more GC-friendly because it uses dynamic progressive block
> scaling internally).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
