[jira] Commented: (LUCENE-2871) Use FileChannel in FSDirectory
[ https://issues.apache.org/jira/browse/LUCENE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12984134#action_12984134 ] Shay Banon commented on LUCENE-2871: Strange, did not get it when running the tests, will try and find out why it can happen. Use FileChannel in FSDirectory -- Key: LUCENE-2871 URL: https://issues.apache.org/jira/browse/LUCENE-2871 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2871.patch Explore using FileChannel in FSDirectory to see if it improves write operations performance -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2871) Use FileChannel in FSDirectory
[ https://issues.apache.org/jira/browse/LUCENE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12984139#action_12984139 ] Michael McCandless commented on LUCENE-2871: Yeah, me neither -- tests all pass when I force dir to eg NIOFSDir, and, my benchmark runs on the 100K index; just fails for the 10M index... curious. Use FileChannel in FSDirectory -- Key: LUCENE-2871 URL: https://issues.apache.org/jira/browse/LUCENE-2871 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2871.patch Explore using FileChannel in FSDirectory to see if it improves write operations performance -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2871) Use FileChannel in FSDirectory
[ https://issues.apache.org/jira/browse/LUCENE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12984145#action_12984145 ] Uwe Schindler commented on LUCENE-2871: --- Looking at the current patch, the class seems wrong. In my opinion, this should be only in NIOFSDirectory. SimpleFSDir should only use RAF. Use FileChannel in FSDirectory -- Key: LUCENE-2871 URL: https://issues.apache.org/jira/browse/LUCENE-2871 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2871.patch Explore using FileChannel in FSDirectory to see if it improves write operations performance -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2871) Use FileChannel in FSDirectory
[ https://issues.apache.org/jira/browse/LUCENE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12984206#action_12984206 ] Shay Banon commented on LUCENE-2871: bq. Looking at the current patch, the class seems wrong. In my opinion, this should be only in NIOFSDirectory. SimpleFSDir should only use RAF. Its a good question, not sure what to do with it. Here is the problem. The channel output can be used with all 3 FS dirs (simple, nio, and mmap), and actually might make sense to be used even with SimpleFS (i.e. using non nio to read, but use file channel to write). In order to have all of them supported, currently, the simplest way is to put it in the base class so code will be shared. On IRC, a discussion was made to externalize the outputs and inputs, and then one can more easily pick and choose, but I think this will belong on a different patch. Use FileChannel in FSDirectory -- Key: LUCENE-2871 URL: https://issues.apache.org/jira/browse/LUCENE-2871 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2871.patch, LUCENE-2871.patch Explore using FileChannel in FSDirectory to see if it improves write operations performance -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2871) Use FileChannel in FSDirectory
[ https://issues.apache.org/jira/browse/LUCENE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12984222#action_12984222 ] Earwin Burrfoot commented on LUCENE-2871: - Before arguing where to put this new IndexOutput, I think it's wise to have a benchmark proving we need it at all. I have serious doubts FileChannel's going to outperform RAF.write(). Why should it? And for the purporses of benchmark it can be anywhere. Use FileChannel in FSDirectory -- Key: LUCENE-2871 URL: https://issues.apache.org/jira/browse/LUCENE-2871 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2871.patch, LUCENE-2871.patch Explore using FileChannel in FSDirectory to see if it improves write operations performance -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2871) Use FileChannel in FSDirectory
[ https://issues.apache.org/jira/browse/LUCENE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12984263#action_12984263 ] Shay Banon commented on LUCENE-2871: Agreed Earwin, lets first see if it make sense, this is just an experiment and might not make sense for single threaded writes. Use FileChannel in FSDirectory -- Key: LUCENE-2871 URL: https://issues.apache.org/jira/browse/LUCENE-2871 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2871.patch, LUCENE-2871.patch Explore using FileChannel in FSDirectory to see if it improves write operations performance -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2871) Use FileChannel in FSDirectory
[ https://issues.apache.org/jira/browse/LUCENE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12984373#action_12984373 ] Michael McCandless commented on LUCENE-2871: OK -- I was able to index 10M docs w/ the new patch. And search results are identical. But the indexing time on trunk vs the patch were nearly identical -- 536.80 sec (trunk) and 536.06 (w/ patch). But, this is on a fast machine, lots of RAM (so writes go straight to buffer cache) and an SSD, using 6 indexing threads. Use FileChannel in FSDirectory -- Key: LUCENE-2871 URL: https://issues.apache.org/jira/browse/LUCENE-2871 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2871.patch, LUCENE-2871.patch Explore using FileChannel in FSDirectory to see if it improves write operations performance -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2871) Use FileChannel in FSDirectory
[
https://issues.apache.org/jira/browse/LUCENE-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12983798#action_12983798
]
Michael McCandless commented on LUCENE-2871:
I attempted to benchmark this patch, testing indexing time for Wikipedia docs,
but I hit this exception:
{noformat}
Exception in thread Lucene Merge Thread #49
org.apache.lucene.index.MergePolicy$MergeException:
java.nio.BufferOverflowException
at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:507)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:472)
Caused by: java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Buffer.java:501)
at java.nio.DirectByteBuffer.putLong(DirectByteBuffer.java:745)
at
org.apache.lucene.store.FSDirectory$FSChannelIndexOutput.writeLong(FSDirectory.java:706)
at
org.apache.lucene.index.codecs.PrefixCodedTermsWriter.writeTrailer(PrefixCodedTermsWriter.java:146)
at
org.apache.lucene.index.codecs.PrefixCodedTermsWriter.close(PrefixCodedTermsWriter.java:129)
at
org.apache.lucene.index.PerFieldCodecWrapper$FieldsWriter.close(PerFieldCodecWrapper.java:80)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:559)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:120)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3084)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2768)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:379)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:447)
{noformat}
This was after adding ~6 M docs... the 100K doc index built just fine. Shay
any ideas?
Use FileChannel in FSDirectory
--
Key: LUCENE-2871
URL: https://issues.apache.org/jira/browse/LUCENE-2871
Project: Lucene - Java
Issue Type: New Feature
Components: Store
Reporter: Shay Banon
Attachments: LUCENE-2871.patch
Explore using FileChannel in FSDirectory to see if it improves write
operations performance
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
