[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: LUCENE-550.patch classdiagram.png Some dull colors, rendered via PDF to PNG and then scaled to fit 1024x768. Also softscaled in package.html, but linked to when clicked on. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: BinarySearchUtils.Apache.java, classdiagram.png, > HitCollectionBench.jpg, LUCENE-550.patch, LUCENE-550.patch, LUCENE-550.patch, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > Represented as a coupled graph of class instances, this all-in-memory index > store implementation delivers search results up to a 100 times faster than > the file-centric RAMDirectory at the cost of greater RAM consumption. > Performance seems to be a little bit better than log2n (binary search). No > real data on that, just my eyes. > Populated with a single document InstantiatedIndex is almost, but not quite, > as fast as MemoryIndex. > At 20,000 document 10-50 characters long InstantiatedIndex outperforms > RAMDirectory some 30x, > 15x at 100 documents of 2000 charachters length, > and is linear to RAMDirectory at 10,000 documents of 2000 characters length. > Mileage may vary depending on term saturation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: classdiagram.jpg) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: BinarySearchUtils.Apache.java, HitCollectionBench.jpg, > LUCENE-550.patch, LUCENE-550.patch, LUCENE-550_20071021_no_core_changes.txt, > test-reports.zip > > > Represented as a coupled graph of class instances, this all-in-memory index > store implementation delivers search results up to a 100 times faster than > the file-centric RAMDirectory at the cost of greater RAM consumption. > Performance seems to be a little bit better than log2n (binary search). No > real data on that, just my eyes. > Populated with a single document InstantiatedIndex is almost, but not quite, > as fast as MemoryIndex. > At 20,000 document 10-50 characters long InstantiatedIndex outperforms > RAMDirectory some 30x, > 15x at 100 documents of 2000 charachters length, > and is linear to RAMDirectory at 10,000 documents of 2000 characters length. > Mileage may vary depending on term saturation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: classdiagram.jpg LUCENE-550.patch Added more javadocs. The patch is not sticky enough for instantiated/docs/classdiagram.jpg. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: BinarySearchUtils.Apache.java, classdiagram.jpg, > HitCollectionBench.jpg, LUCENE-550.patch, LUCENE-550.patch, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > Represented as a coupled graph of class instances, this all-in-memory index > store implementation delivers search results up to a 100 times faster than > the file-centric RAMDirectory at the cost of greater RAM consumption. > Performance seems to be a little bit better than log2n (binary search). No > real data on that, just my eyes. > Populated with a single document InstantiatedIndex is almost, but not quite, > as fast as MemoryIndex. > At 20,000 document 10-50 characters long InstantiatedIndex outperforms > RAMDirectory some 30x, > 15x at 100 documents of 2000 charachters length, > and is linear to RAMDirectory at 10,000 documents of 2000 characters length. > Mileage may vary depending on term saturation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Ingersoll updated LUCENE-550: --- Attachment: LUCENE-550.patch Cleaned up a few things, added CHANGES.txt, added ASL to a file.I'll commit tomorrow, pending any more feedback. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: BinarySearchUtils.Apache.java, HitCollectionBench.jpg, > LUCENE-550.patch, LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > Represented as a coupled graph of class instances, this all-in-memory index > store implementation delivers search results up to a 100 times faster than > the file-centric RAMDirectory at the cost of greater RAM consumption. > Performance seems to be a little bit better than log2n (binary search). No > real data on that, just my eyes. > Populated with a single document InstantiatedIndex is almost, but not quite, > as fast as MemoryIndex. > At 20,000 document 10-50 characters long InstantiatedIndex outperforms > RAMDirectory some 30x, > 15x at 100 documents of 2000 charachters length, > and is linear to RAMDirectory at 10,000 documents of 2000 characters length. > Mileage may vary depending on term saturation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olivier Chafik updated LUCENE-550: -- Attachment: BinarySearchUtils.Apache.java Here is an enhanced binarySearch method for int arrays, which I wrote and wish to donate to the ASF (for the Lucene project or any other purpose), following Karl Wettin's request. This code was initially published on my blog : http://ochafik.free.fr/blog/?p=106 Have fun with it ! -- Olivier Chafik > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: BinarySearchUtils.Apache.java, HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > Represented as a coupled graph of class instances, this all-in-memory index > store implementation delivers search results up to a 100 times faster than > the file-centric RAMDirectory at the cost of greater RAM consumption. > Performance seems to be a little bit better than log2n (binary search). No > real data on that, just my eyes. > Populated with a single document InstantiatedIndex is almost, but not quite, > as fast as MemoryIndex. > At 20,000 document 10-50 characters long InstantiatedIndex outperforms > RAMDirectory some 30x, > 15x at 100 documents of 2000 charachters length, > and is linear to RAMDirectory at 10,000 documents of 2000 characters length. > Mileage may vary depending on term saturation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Description: Represented as a coupled graph of class instances, this all-in-memory index store implementation delivers search results up to a 100 times faster than the file-centric RAMDirectory at the cost of greater RAM consumption. Performance seems to be a little bit better than log2n (binary search). No real data on that, just my eyes. Populated with a single document InstantiatedIndex is almost, but not quite, as fast as MemoryIndex. At 20,000 document 10-50 characters long InstantiatedIndex outperforms RAMDirectory some 30x, 15x at 100 documents of 2000 charachters length, and is linear to RAMDirectory at 10,000 documents of 2000 characters length. Mileage may vary depending on term saturation. was: An non file centrinc all in memory index. Consumes some 2x the memory of a RAMDirectory (in a term satured index) but is between 3x-60x faster depending on application and how one counts. Average query is about 8x faster. IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and InterfaceIndexModifier. InstantiatedIndex is wrapped in a new top layer index facade (class Index) that comes with factory methods for writers, readers and searchers for unison index handeling. There are decorators with notification handling that can be used for automatically syncronizing searchers on updates, et.c. Index also comes with FS/RAMDirectory implementation. Lucene Fields: [Patch Available] > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > Represented as a coupled graph of class instances, this all-in-memory index > store implementation delivers search results up to a 100 times faster than > the file-centric RAMDirectory at the cost of greater RAM consumption. > Performance seems to be a little bit better than log2n (binary search). No > real data on that, just my eyes. > Populated with a single document InstantiatedIndex is almost, but not quite, > as fast as MemoryIndex. > At 20,000 document 10-50 characters long InstantiatedIndex outperforms > RAMDirectory some 30x, > 15x at 100 documents of 2000 charachters length, > and is linear to RAMDirectory at 10,000 documents of 2000 characters length. > Mileage may vary depending on term saturation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: LUCENE-550_20071019_no_core_changes.txt) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: LUCENE-550_20071017_no_core_changes.txt) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: LUCENE-550_20071008_no_core_changes.txt) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: LUCENE-550_20070817_no_core_changes.txt) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: LUCENE-550_20070928_no_core_changes.txt) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: LUCENE-550_20070808_no_core_changes.txt) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: LUCENE-550_20070804_no_core_changes.txt) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20070808_no_core_changes.txt, > LUCENE-550_20070817_no_core_changes.txt, > LUCENE-550_20070928_no_core_changes.txt, > LUCENE-550_20071008_no_core_changes.txt, > LUCENE-550_20071017_no_core_changes.txt, > LUCENE-550_20071019_no_core_changes.txt, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: lucene-550.jpg) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, > LUCENE-550_20070808_no_core_changes.txt, > LUCENE-550_20070817_no_core_changes.txt, > LUCENE-550_20070928_no_core_changes.txt, > LUCENE-550_20071008_no_core_changes.txt, > LUCENE-550_20071017_no_core_changes.txt, > LUCENE-550_20071019_no_core_changes.txt, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: LUCENE-550_20071021_no_core_changes.txt In this patch: * Replaced all List with T[] as Array.binarySearch is 20% faster than Collections.binarySearch. * Ad hoc binarySearch using variable pivot increase speed of TermDocs.skipTo 20%-400%, courtesy of Olivier Chafik. * Default InstantiatedWriter.mergeFactor changed from 1 to 2500 ;-) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > LUCENE-550_20070804_no_core_changes.txt, > LUCENE-550_20070808_no_core_changes.txt, > LUCENE-550_20070817_no_core_changes.txt, > LUCENE-550_20070928_no_core_changes.txt, > LUCENE-550_20071008_no_core_changes.txt, > LUCENE-550_20071017_no_core_changes.txt, > LUCENE-550_20071019_no_core_changes.txt, > LUCENE-550_20071021_no_core_changes.txt, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: LUCENE-550_20071019_no_core_changes.txt In this patch: * IndexReader.terms(Term) optimization, initial seek now jit-call away given the term exists, rather than using binary search. * A handful of minor optimizations * IndexReader.version() mimics Segment-dito > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > LUCENE-550_20070804_no_core_changes.txt, > LUCENE-550_20070808_no_core_changes.txt, > LUCENE-550_20070817_no_core_changes.txt, > LUCENE-550_20070928_no_core_changes.txt, > LUCENE-550_20071008_no_core_changes.txt, > LUCENE-550_20071017_no_core_changes.txt, > LUCENE-550_20071019_no_core_changes.txt, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: LUCENE-550_20071017_no_core_changes.txt In this path: * As the Segment-dito, non-mapper term vector methods returns null rather than throwing NPE when term vector is not available. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > LUCENE-550_20070804_no_core_changes.txt, > LUCENE-550_20070808_no_core_changes.txt, > LUCENE-550_20070817_no_core_changes.txt, > LUCENE-550_20070928_no_core_changes.txt, > LUCENE-550_20071008_no_core_changes.txt, > LUCENE-550_20071017_no_core_changes.txt, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: LUCENE-550_20071008_no_core_changes.txt In this patch: * Minor discrepancy in IndexReader#norms(String field, byte[] bytes, int offset) between SegmentReader and InstantiatedIndexReader fixed and demonstrated in TestIndicesEquals. http://www.nabble.com/norms%28String-field%2C-byte---bytes%2C-int-offset%29-tf4580460.html#a13075367 * Updated maven pom and fixed some typos in documentation. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > LUCENE-550_20070804_no_core_changes.txt, > LUCENE-550_20070808_no_core_changes.txt, > LUCENE-550_20070817_no_core_changes.txt, > LUCENE-550_20070928_no_core_changes.txt, > LUCENE-550_20071008_no_core_changes.txt, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[
https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wettin updated LUCENE-550:
---
Attachment: LUCENE-550_20070928_no_core_changes.txt
New in this patch:
* Payloads added to TestIndicesEquals
* Package level java docs with UMLet class diagram
* Some additional todo-tags in the code that shows what can be improved
I've noticed that there are some differences in the behavior of IndexWriter and
InstantiatedIndexWriter when a document containing multiple fields with the
same name but different settings, such as:
{code}
d.add(new Field("f", " All work and no play makes Jack a dull boy",
Field.Store.YES, Field.Index.TOKENIZED,
Field.TermVector.WITH_POSITIONS_OFFSETS));
d.add(new Field("f", " All work and no play makes Jack a dull boy",
Field.Store.NO));
d.add(new Field("f", " All work and no play makes Jack a dull boy",
Field.Store.YES, Field.Index.NO_NORMS, Field.TermVector.NO));
{code}
Would this be considered an invalid document? Should there be a term vector or
not? Or perhaps just term vector for the tokens in the first field?
> InstantiatedIndex - faster but memory consuming index
> -
>
> Key: LUCENE-550
> URL: https://issues.apache.org/jira/browse/LUCENE-550
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
>Affects Versions: 2.0.0
>Reporter: Karl Wettin
>Assignee: Grant Ingersoll
> Attachments: HitCollectionBench.jpg, lucene-550.jpg,
> LUCENE-550_20070804_no_core_changes.txt,
> LUCENE-550_20070808_no_core_changes.txt,
> LUCENE-550_20070817_no_core_changes.txt,
> LUCENE-550_20070928_no_core_changes.txt, test-reports.zip, trunk.diff.bz2,
> trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2,
> trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2,
> trunk.diff.bz2, trunk.diff.bz2
>
>
> An non file centrinc all in memory index. Consumes some 2x the memory of a
> RAMDirectory (in a term satured index) but is between 3x-60x faster depending
> on application and how one counts. Average query is about 8x faster.
> IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and
> InterfaceIndexModifier.
> InstantiatedIndex is wrapped in a new top layer index facade (class Index)
> that comes with factory methods for writers, readers and searchers for unison
> index handeling. There are decorators with notification handling that can be
> used for automatically syncronizing searchers on updates, et.c.
> Index also comes with FS/RAMDirectory implementation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: LUCENE-550_20070817_no_core_changes.txt Previously mentioned problems deloused. The phrase (term position) problem turned out to be the constructor InstantiatedIndex(IndexReader) that had a bug, ending up with a index not equal to one created via InstantiatedIndexWriter. I also did a bunch of tests on how much it would speed up by replacing the binary searches over lists with hash tables (maps). Gained perhaps 5% speed, but lost quite a bit of RAM, so I reverted those things. Do you want more test cases than the TestIndicesEquals? Payloads needs to be verified. I never really worked with them, and the Directory-centric test will not be ported easily. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > LUCENE-550_20070804_no_core_changes.txt, > LUCENE-550_20070808_no_core_changes.txt, > LUCENE-550_20070817_no_core_changes.txt, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: LUCENE-550_20070808_no_core_changes.txt Added support for payloads Reintroduced InstantiatedIndexWriter (no locks!) Reintroduced TestIndicesEquals Introduced build.xml Introduced pom.xml (this file is missing java 1.5 setting) Added some silly javadocs It also hit me that I could have a HashMap parallell to the List orderdTerms. The latter is currently beeing binarysearched in TermEnum, and a HashMap would make it much faster, especially as the index grows. Might speed things up alot. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Grant Ingersoll > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > LUCENE-550_20070804_no_core_changes.txt, > LUCENE-550_20070808_no_core_changes.txt, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: LUCENE-550_20070804_no_core_changes.txt This is a small and completely isolated version of InstantiatedIndex, the results of my "last attempt" thread: http://www.nabble.com/Last-attempt-tf4153815.html It requires no changes to the Lucene core but hogs a bit more RAM and probably depends on your JIT to avoid wasting CPU. So prior required definalization and generalization is replaced by aggregation (strategy pattern). I also had to remove all the polymorphic index handling (IndexWriterInterface et c), and I have removed the IndexWriter in InstantiatedIndex. One now have to create a new InstantiatedIndex and pass down an IndexReader instead. So there is no appending allowed. Also, there are no locks no more, but that should not be needed anymore. The port of the complete test suite from Lucene to the unison index handling has been removed. Ie there are no real test cases that demonstrate this patch. Anything but term vectors and payloads should work great though. The code base is over a year old and these are new features I did not have time to implement or test. No new benchmarks. The greatest loss is the loss of features, not CPU and RAM. Perhaps it waste 15% more resources than the previous patch? As I personally enjoy the features removed in this patch, I will keep on running Lucene 2.0 and the old version, but this should be easier to understand and maintain if anyone else wants to take a look at it. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin >Assignee: Karl Wettin > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > LUCENE-550_20070804_no_core_changes.txt, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: HitCollectionBench.jpg) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > test-reports.zip, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: HitCollectionBench.jpg x/y axis names updates > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > test-reports.zip, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: HitCollectionBench.jpg) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > test-reports.zip, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: HitCollectionBench.jpg made graph more readable > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > test-reports.zip, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: HitCollectionBench.jpg A graph showing performance of hit collection using InstantiatedIndex, RAMDirectory and FSDirectory. In essence, there is no great win in pure search time when there are more than 7000 documents. However, retreiving documents is still not associate with any cost what so ever, so in a 25 sized index that use Lucene for persistency of fields, I still see a boost with 6-10x or so compared to RAMDirectory. documents in corpus \t queries per second [EMAIL PROTECTED] 250 37530,00 500 29610,00 750 22612,50 100019267,50 125016027,50 150014737,50 175013230,00 200012322,50 225011482,50 250010125,00 27509802,50 30008508,25 32508469,80 35007788,61 37505207,29 40005484,52 42504912,50 45004420,58 47504006,49 50004357,50 52503886,67 55003573,93 57503236,76 60003602,10 62503420,00 65003075,00 67502805,00 70002680,98 72502908,55 75002769,46 77502644,86 80002496,25 82502377,50 85002578,71 87502390,11 90002160,00 92502037,96 95001872,19 97502041,38 1 1959,12 Created 1 documents [EMAIL PROTECTED] 250 4845,00 500 3986,01 750 4330,67 10004682,82 12504148,78 15004847,65 17504535,23 20004192,50 22504203,30 25003695,65 27503742,50 30003485,76 32503470,76 35003525,00 37502877,61 40003221,78 42502983,51 45002982,02 47502724,55 50003092,86 52502646,18 55002940,00 57502709,58 60002423,30 62502602,50 65002305,39 67502462,57 70001815,00 72502431,42 75002171,74 77502297,90 80002134,30 82502308,85 85002038,98 87502231,65 90002097,90 92502041,38 95001819,77 97502102,24 1 1876,87 Created 1 documents [EMAIL PROTECTED] 250 3448,28 500 2422,50 750 2677,50 10002607,39 12502241,92 15002486,27 17502472,53 20001733,52 22502325,00 25002194,21 27501969,55 30002125,75 32502009,00 35001473,08 37501858,14 40001925,57 42501671,66 45001786,25 47501694,15 50001217,63 52501595,11 55001745,75 57501526,18 60001431,78 62501524,66 65001648,35 67501544,23 70001428,22 72501487,29 75001494,02 77501106,13 80001455,00 82501284,86 85001182,63 87501292,33 90001399,70 92501000,00 95001291,04 97501359,56 1 1194,62 Created 1 documents > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: HitCollectionBench.jpg, lucene-550.jpg, > test-reports.zip, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automat
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 Patched contrib/benchmark to support InstantiatedIndex. Fixed a bug with mergeFactor. Reverted java 1.5 changes in PriorityQueue to (ClassCasting). (This is actually a spell checker thingy, but due to local dependencies the changes are located in this patch). Removed write locks. These had severe bugs and need to be reconsidered. Should be back in next patch. Using multiple InstantiatedIndex:es as segments on a MultiReader rather than updating the same index, this can be made completly lockless. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: lucene-550.jpg, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: didyoumean.jpg) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: lucene-550.jpg, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 Removed the dependencies to LUCENE-626. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: lucene-550.jpg, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 Switched from java.util.PriorityQueue to org.apache.lucene.util.PriorityQueue, and made the latter . Fixed some major bugs in the TermFreqVector inspection for the spell checker. Demonstrate in TestGoalJuror how to build an a priori corpus for the ngram token suggester based on user input by inverting the suggestion dictionary. That should probably be extracted to a helper class in the future. This makes it faster to query the a apriori, but it also means that what the system takes for grantent is correct comes from user input, and even if the correct data is what users point out as a real query goal, it does not have to be correct. Although, it makes the suggester much faster. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: didyoumean.jpg, lucene-550.jpg, test-reports.zip, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 New Patch. Mainly updates in contrib/didyoumean. Merged some core conflicts. TestGoalJuror now import 200,000 real user queries from a log containing session id, query, category, timestamp and number of hits, ordered by session id and time. This means that the trainer and suggester are not aware of if the user followed or ignored a suggestion from the system, what results was inspected, if the query contained a goal, et c. So it does not work as if trained from the start with the adaptive layer. Still, the suggester navigates the dictionary fairly well and misspelled queries will be suggested the correct suggestion, but many correct spelled phrases will recommend something silly. As one start reporting user interaction to the suggester any silly recommendation should go away. In essence, it can only adapt the suggestions positive based on what the QueryGoalJuror says is a goal. Negative is only when a user don't take a suggestion. It could be solved with bootstrapping. Will mess with that later. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: didyoumean.jpg, lucene-550.jpg, test-reports.zip, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 Support for deleteDocuments in IndexWriterInterface, InstantiatedIndex and NotifiableIndex. Somewhat hacky solution to pick up the deletions in NotifiableIndex, but it is a solution. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: didyoumean.jpg, lucene-550.jpg, test-reports.zip, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 Added lots of documentation > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: didyoumean.jpg, lucene-550.jpg, test-reports.zip, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Comment: was deleted > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: didyoumean.jpg, lucene-550.jpg, test-reports.zip, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: didyoumean.jpg UML class diagram of the adaptive spell checker with all java docs as comments > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: didyoumean.jpg, lucene-550.jpg, test-reports.zip, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 Updated spell checker code > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: lucene-550.jpg, test-reports.zip, trunk.diff.bz2, > trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[
https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wettin updated LUCENE-550:
---
Attachment: trunk.diff.bz2
Introduced a method in instantiated index that appends the entire content to
any other index.
{code}
/**
* Adds the complete content of this instantiated index on to any other index
using an index writer.
*
* This can for instance be used for
* merging multiple instantiated indices
* and periodically storing persistent snapshots in an FSDirectory.
*
* Non stored offsets are partially rebuilt. This can be improved quite a
bit. See comments in code.
*
* The analyzer creates one complete token stream of all fields with the same
name the first time it is requested,
* and after that an empty for each remaining. todo: this is a problem?
*
* It can be buggy if the same token appears as synonym to it self (position
increment 0). not really something to worry about.. or?
*
* @param indexWriter represents the index on wich to add all the content of
this instantiated index.
* @throws IOException when accessing parameter indexWriter
*/
public void writeToIndex(IndexWriterInterface indexWriter) throws IOException
{
{code}
> InstantiatedIndex - faster but memory consuming index
> -
>
> Key: LUCENE-550
> URL: https://issues.apache.org/jira/browse/LUCENE-550
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
>Affects Versions: 2.0.0
>Reporter: Karl Wettin
> Assigned To: Karl Wettin
> Attachments: lucene-550.jpg, test-reports.zip, trunk.diff.bz2,
> trunk.diff.bz2, trunk.diff.bz2, trunk.diff.bz2
>
>
> An non file centrinc all in memory index. Consumes some 2x the memory of a
> RAMDirectory (in a term satured index) but is between 3x-60x faster depending
> on application and how one counts. Average query is about 8x faster.
> IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and
> InterfaceIndexModifier.
> InstantiatedIndex is wrapped in a new top layer index facade (class Index)
> that comes with factory methods for writers, readers and searchers for unison
> index handeling. There are decorators with notification handling that can be
> used for automatically syncronizing searchers on updates, et.c.
> Index also comes with FS/RAMDirectory implementation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[
https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wettin updated LUCENE-550:
---
Attachment: trunk.diff.bz2
Can now be loaded from, and be persisted in an FSDirectory.
The actual implementation is a bit more abtract than that though. It is not
super nice yet, but all low level index comparator tests pass.
Introduced functionallity to load an instantiated from any index reader (e.g. a
FSDirectory)
{code:java}
/**
* Creates a new instantiated index that looks just like the index in a
specific state as represented by a reader.
*
* @param sourceIndexReader the source index this new instantiated index will
be copied from.
* @throws IOException if the source index is not optimized, or when accesing
the source.
*/
public InstantiatedIndex(IndexReader sourceIndexReader) throws IOException {
{code}
Also introduced class SimpleSychronizedIndex, a class that kind of works like
unix command "tee", makes sure that all changes to a main index (e.g. an
instantiated index) also is applied to a mirror index (e.g. the fs directory
loaded to the instantiated index at constructor time).
Some class that handles these two things a single entity will probably be added
soon.
Basiacally this is replicating changes to a secondary index on commits. Thus it
takes about twice the time to insert documents. Perhaps the secondary index
should be updated in a secondary thread?
> InstantiatedIndex - faster but memory consuming index
> -
>
> Key: LUCENE-550
> URL: https://issues.apache.org/jira/browse/LUCENE-550
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
>Affects Versions: 2.0.0
>Reporter: Karl Wettin
> Assigned To: Karl Wettin
> Attachments: lucene-550.jpg, test-reports.zip, trunk.diff.bz2,
> trunk.diff.bz2, trunk.diff.bz2
>
>
> An non file centrinc all in memory index. Consumes some 2x the memory of a
> RAMDirectory (in a term satured index) but is between 3x-60x faster depending
> on application and how one counts. Average query is about 8x faster.
> IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and
> InterfaceIndexModifier.
> InstantiatedIndex is wrapped in a new top layer index facade (class Index)
> that comes with factory methods for writers, readers and searchers for unison
> index handeling. There are decorators with notification handling that can be
> used for automatically syncronizing searchers on updates, et.c.
> Index also comes with FS/RAMDirectory implementation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[
https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karl Wettin updated LUCENE-550:
---
Attachment: trunk.diff.bz2
Added support for contrib/memory MemoryIndex, so now it works with readers and
writers as if it was any other index.
Added a consumer level index implementation that handles cache, notifications,
and all the stuff this issue is about:
// This is the instace one is supposed to use for all access against the index
in this JVM.
IndexFacade index = new IndexFacade(new RAMDirectoryIndex());
// Accessors
IndexWriterInterface writer = index.indexWriterFactory(anayzler, true);
Document doc = new Document();
doc.add(...
writer.add(doc);
writer.close();
IndexReader deleter = index.indexReaderFactory();
index.getSearcher().search(...
index.getReader().doc(0)
deleter.close();
assertEquals(0, index.getReader().numDocs());
public class IndexFacade {
/** wrapps any storage, optional cache settings */
public IndexFacade(I index, CachedSearcher.HitCollectionCacheState
hitCollectionCache, boolean topDocsCache, boolean topFieldsCache, boolean
documentsCache) throws IOException {
public CachedSearcher getSearcher() throws IOException {
/** The general consumer searcher to be used when querying this index. Always
fresh. */
public Searcher getSearcher() throws IOException {
/** The general consumer read only index reader to be used when inspecting
this index. Always fresh. */
public IndexReader getReader() throws IOException {
> InstantiatedIndex - faster but memory consuming index
> -
>
> Key: LUCENE-550
> URL: https://issues.apache.org/jira/browse/LUCENE-550
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
>Affects Versions: 2.0.0
>Reporter: Karl Wettin
> Assigned To: Karl Wettin
> Attachments: lucene-550.jpg, test-reports.zip, trunk.diff.bz2,
> trunk.diff.bz2
>
>
> An non file centrinc all in memory index. Consumes some 2x the memory of a
> RAMDirectory (in a term satured index) but is between 3x-60x faster depending
> on application and how one counts. Average query is about 8x faster.
> IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and
> InterfaceIndexModifier.
> InstantiatedIndex is wrapped in a new top layer index facade (class Index)
> that comes with factory methods for writers, readers and searchers for unison
> index handeling. There are decorators with notification handling that can be
> used for automatically syncronizing searchers on updates, et.c.
> Index also comes with FS/RAMDirectory implementation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 Refactored the Term->Document relationships a bit for speed optimizations. It also resulted with getting all term frequency vector information except for offsets free of charge. More information on that in the class diagram. Removed a whole bunch of todo:s in the writer and reader. The current lock implementen is worthless. I need to read up on RentrentLock. Or should I perhaps use the lock Directory:s use? (And that class diagram is of course granted for ASF, my misstake.) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: lucene-550.jpg, test-reports.zip, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: lucene-550.jpg, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: issue550.jpg) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: lucene-550.jpg, test-reports.zip > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: lucene-550.jpg new diagram with lots of notes (this is also available in the patch as an uxf-file for umlet) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: issue550.jpg, lucene-550.jpg, test-reports.zip, > trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: issue550.jpg, test-reports.zip, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: issue550.jpg, test-reports.zip, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 Patch of the week. Changes: * CachedSearcher -- soft referenced hit collection-, TopDocs- and TopFieldDocs cache. Backed by NotifiableIndex. Removed Hits cache due to uncertainty but introduced: * CachedIndexReader -- soft referenced documents cache. Backed by NotifiableIndex. TopDocs/TopFieldDocs- and IndexReader cache combined almost replace a fully cached Hits. The number of unit tests and detail of them is increasing. The plan is now to have the cached reader pre-loading documents to memory from an own thread when server load allows it. Also added some abstractation levers used by above: * AutofreshedIndexReader -- always up to date with the index. * ReadOnlyIndexReader -- makes sure the user don't delete stuff with the decorated reader. Had some problems with decorating the IndexModifierInterface against Directory in NotifiableIndex, so removed the Index.indexModifierFactory() and introduced a index facade backed version: org.apache.lucene.index.facade.IndexModifier(myIndex, analyzer, create) where all reader/writer creation is myIndex.indexReaderFactory() and indexWriterFactory(); Makes the Notifiable code a bit simpler. > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: issue550.jpg, test-reports.zip, trunk.diff, > trunk.diff.bz2, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: issue550.jpg > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: issue550.jpg, test-reports.zip, trunk.diff, > trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: lucene.1.9-karl1.jpg) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: test-reports.zip, trunk.diff, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: class_diagram.png) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: test-reports.zip, trunk.diff, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: class_diagram.png) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: test-reports.zip, trunk.diff, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: test-reports.zip, trunk.diff, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, test-reports.zip, trunk.diff, trunk.diff.bz2, > trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated LUCENE-550: Attachment: (was: trunk.diff) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, test-reports.zip, trunk.diff, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated LUCENE-550: Attachment: trunk.diff test-reports.zip > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, test-reports.zip, trunk.diff, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated LUCENE-550: Attachment: (was: test-reports.zip) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, test-reports.zip, trunk.diff, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated LUCENE-550: Attachment: trunk.diff test-reports.zip > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, test-reports.zip, trunk.diff, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: trunk.diff.bz2) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: (was: lucene2karl-061122.tar.gz) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ https://issues.apache.org/jira/browse/LUCENE-550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karl Wettin updated LUCENE-550: --- Attachment: trunk.diff.bz2 > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: https://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, trunk.diff.bz2 > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ http://issues.apache.org/jira/browse/LUCENE-550?page=all ] Karl Wettin updated LUCENE-550: --- Attachment: (was: InstanciatedIndexTermEnum.java) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: http://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, lucene2karl-061122.tar.gz > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ http://issues.apache.org/jira/browse/LUCENE-550?page=all ] Karl Wettin updated LUCENE-550: --- Attachment: (was: instanciated_20060527.tar) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: http://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, lucene2karl-061122.tar.gz > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ http://issues.apache.org/jira/browse/LUCENE-550?page=all ] Karl Wettin updated LUCENE-550: --- Attachment: (was: lucene2-karl_20060722.tar.gz) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: http://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, lucene2karl-061122.tar.gz > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ http://issues.apache.org/jira/browse/LUCENE-550?page=all ] Karl Wettin updated LUCENE-550: --- Attachment: (was: lucene2-karl_20060723.tar.gz) > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: http://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > lucene.1.9-karl1.jpg, lucene2karl-061122.tar.gz > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-550) InstantiatedIndex - faster but memory consuming index
[ http://issues.apache.org/jira/browse/LUCENE-550?page=all ] Karl Wettin updated LUCENE-550: --- Summary: InstantiatedIndex - faster but memory consuming index (was: InstanciatedIndex - faster but memory consuming index) Affects Version/s: 2.0.0 (was: 1.9) Description: An non file centrinc all in memory index. Consumes some 2x the memory of a RAMDirectory (in a term satured index) but is between 3x-60x faster depending on application and how one counts. Average query is about 8x faster. IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and InterfaceIndexModifier. InstantiatedIndex is wrapped in a new top layer index facade (class Index) that comes with factory methods for writers, readers and searchers for unison index handeling. There are decorators with notification handling that can be used for automatically syncronizing searchers on updates, et.c. Index also comes with FS/RAMDirectory implementation. was: After fixing the bugs, it's now 4.5 -> 5 times the speed. This is true for both at index and query time. Sorry if I got your hopes up too much. There are still things to be done though. Might not have time to do anything with this until next month, so here is the code if anyone wants a peek. Not good enough for Jira yet, but if someone wants to fool around with it, here it is. The implementation passes a TermEnum -> TermDocs -> Fields -> TermVector comparation against the same data in a Directory. When it comes to features, offsets don't exists and positions are stored ugly and has bugs. You might notice that norms are float[] and not byte[]. That is me who refactored it to see if it would do any good. Bit shifting don't take many ticks, so I might just revert that. I belive the code is quite self explaining. InstanciatedIndex ii = .. ii.new InstanciatedIndexReader(); ii.addDocument(s).. replace IndexWriter for now. Assignee: Karl Wettin > InstantiatedIndex - faster but memory consuming index > - > > Key: LUCENE-550 > URL: http://issues.apache.org/jira/browse/LUCENE-550 > Project: Lucene - Java > Issue Type: New Feature > Components: Store >Affects Versions: 2.0.0 >Reporter: Karl Wettin > Assigned To: Karl Wettin > Attachments: class_diagram.png, class_diagram.png, > instanciated_20060527.tar, InstanciatedIndexTermEnum.java, > lucene.1.9-karl1.jpg, lucene2-karl_20060722.tar.gz, > lucene2-karl_20060723.tar.gz, lucene2karl-061122.tar.gz > > > An non file centrinc all in memory index. Consumes some 2x the memory of a > RAMDirectory (in a term satured index) but is between 3x-60x faster depending > on application and how one counts. Average query is about 8x faster. > IndexWriter and IndexModifier have been realized in InterfaceIndexWriter and > InterfaceIndexModifier. > InstantiatedIndex is wrapped in a new top layer index facade (class Index) > that comes with factory methods for writers, readers and searchers for unison > index handeling. There are decorators with notification handling that can be > used for automatically syncronizing searchers on updates, et.c. > Index also comes with FS/RAMDirectory implementation. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
