[
https://issues.apache.org/jira/browse/LUCENE-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609827#action_12609827
]
Michael McCandless commented on LUCENE-753:
-------------------------------------------
{quote}
SeparateFile (opening a new file descriptor per reader) is not a real
implementation that could be used... it's more of a theoretical maximum IMO.
Also remember that you can't open a new fd on demand since the file might
already be deleted. We would need a real PooledClassicFile implementation (like
PooledPread).
{quote}
True, we'd have to make a real pool, but I'd think we want the sync() to be on
cloning and not on every read. I think Lucene's usage of the open files
(clones are made & used up quickly and closed) would work well with that
approach. I think at this point we should build out an underlying pool and
then test all of these different approaches under the pool.
And yes we cannot just open a new fd on demand if the file has been deleted.
But I'm thinking that may not matter in practice. Ie if the pool wants to open
a new fd, it can attempt to do so, and if the file was deleted it must then
return a shared access wrapper to the fd it already has open. Large segments
are where the contention will be and large segments are not often deleted.
Plus people tend to open new readers if such a large change has taken place to
the index.
{quote}
On non-windows it looks like ChannelPread is probably the right choice.. near
max performance and min fd usage
{quote}
But on Linux I saw 44% speedup for serial=true case with 4 threads using
SeparateFile vs ChannelPread, which I was very surprised by. But then again
it's synthetic so it may not matter in real Lucene searches.
> Use NIO positional read to avoid synchronization in FSIndexInput
> ----------------------------------------------------------------
>
> Key: LUCENE-753
> URL: https://issues.apache.org/jira/browse/LUCENE-753
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
> Reporter: Yonik Seeley
> Attachments: FileReadTest.java, FileReadTest.java, FileReadTest.java,
> FileReadTest.java, FileReadTest.java, FileReadTest.java, FileReadTest.java,
> FSIndexInput.patch, FSIndexInput.patch, lucene-753.patch
>
>
> As suggested by Doug, we could use NIO pread to avoid synchronization on the
> underlying file.
> This could mitigate any MT performance drop caused by reducing the number of
> files in the index format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
