[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12655525#action_12655525 ] Otis Gospodnetic commented on LUCENE-1026: -- My impression was that this didn't stick, so I'd drop it. Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, IndexAccessor-05.27.2008.zip, IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12654258#action_12654258 ] Mark Miller commented on LUCENE-1026: - Anyone use this? I think its convenient myself, but these IndexAccessor patches have never seemed very popular in terms of making it into Lucene. I'd like to get to the point of either dropping the idea and closing the two issues that deal with it, or moving something to contrib. Any thoughts / preferences? Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, IndexAccessor-05.27.2008.zip, IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12600169#action_12600169 ] John Patterson commented on LUCENE-1026: Just checked out the source from the svn link above. I've got a few questions about the code 1 - why use ReentrantLock rather than synchronized blocks? The advice I have come across is to use the standard mechanism unless you need the extra functionality. Can't see anywhere in the code that needs this e.g. obtaining and releasing the lock in different methods or using more than one condition. 2 - why call logger.isLoggable all the time for simple log statements? 3 - In getWriter(boolean) the parameter is ignored if a write already exists. 4 - Why is condition.signalAll() called from getReadingReader()? Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12600266#action_12600266 ] Mark Miller commented on LUCENE-1026: - Bunch of new code coming on this tonight or tomorrow...I consider it basically the 1.0 version. 1. In my testing the ReentrantLock is much faster than the sync blocks under contention. 2. Logger.isLoggable is supposed to be faster than logging at a level thats not turned on. 3. The param is not for create, but for autoCommit. Create is called when you ask for an accessor that doesnt exist yet. 4. Why do you ask? I can't remember at the moment, but at first glance it looks like its unnecessary. It may have been done just because its done in getSearcher (where it is needed). Ill investigate further and possibly take it out. The new code fixes a bunch of issues with the Reader handling. The Searcher handling works great though. Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12600270#action_12600270 ] Mark Miller commented on LUCENE-1026: - Bunch of new code coming on this tonight or tomorrow...I consider it basically the 1.0 version. 1. In my testing the ReentrantLock is much faster than the sync blocks under contention. 2. Logger.isLoggable is supposed to be faster than logging at a level thats not turned on. 3. The param is not for create, but for autoCommit. Create is called when you ask for an accessor that doesnt exist yet. 4. Why do you ask? I can't remember at the moment, but at first glance it looks like its unnecessary. It may have been done just because its done in getSearcher (where it is needed). Ill investigate further and possibly take it out. The new code fixes a bunch of issues with the Reader handling. The Searcher handling works great though. Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[
https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12600274#action_12600274
]
Sean Timm commented on LUCENE-1026:
---
bq. 2. Logger.isLoggable is supposed to be faster than logging at a level thats
not turned on.
That is certainly true if you are creating a string from an object, or
concatenating strings. I don't believe that is true for static strings.
E.g., {code}logger.fine(closing cached reading reader);{code} you might as
well skip the Logger.isLoggable. In the case {code}logger.fine(closing cached
searchers ( + cachedSearchers.size() + ));{code} it is certainly better to
check the log level first.
Provide a simple way to concurrently access a Lucene index from multiple
threads
Key: LUCENE-1026
URL: https://issues.apache.org/jira/browse/LUCENE-1026
Project: Lucene - Java
Issue Type: New Feature
Components: Index, Search
Reporter: Mark Miller
Priority: Minor
Attachments: DefaultIndexAccessor.java,
DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip,
IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip,
IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip,
IndexAccessor.04032008.zip, IndexAccessor.java, IndexAccessor.zip,
IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip,
shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java,
StopWatch.java, TestIndexAccessor.java
For building interactive indexes accessed through a network/internet
(multiple threads).
This builds upon the LuceneIndexAccessor patch. That patch was not very
newbie friendly and did not properly handle MultiSearchers (or at the least
made it easy to get into trouble).
This patch simplifies things and provides out of the box support for sharing
the IndexAccessors across threads. There is also a simple test class and
example SearchServer to get you started.
Future revisions will be zipped.
Works pretty solid as is, but could use the ability to warm new Searchers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
You make a good point. I think I will prob make this change. Asgeir Frimannsson (JIRA) wrote: [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12576858#action_12576858 ] Asgeir Frimannsson commented on LUCENE-1026: Is there any specific reason why this indexaccessor is limited to FSDirectory based indexes? I see FSDirectory.getFile() is used as a unique key in the list of IndexAccessors in the factory. However, it seems more natural to use dir.getLockID() for this purpose. Then it would be possible to use a generic Directory rather than the file-system specific FSDirectory. Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, IndexAccessor.java, IndexAccessor.zip, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12576858#action_12576858 ] Asgeir Frimannsson commented on LUCENE-1026: Is there any specific reason why this indexaccessor is limited to FSDirectory based indexes? I see FSDirectory.getFile() is used as a unique key in the list of IndexAccessors in the factory. However, it seems more natural to use dir.getLockID() for this purpose. Then it would be possible to use a generic Directory rather than the file-system specific FSDirectory. Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor-02.04.2008.zip, IndexAccessor-02.07.2008.zip, IndexAccessor-02.28.2008.zip, IndexAccessor-1.26.2008.zip, IndexAccessor-2.15.2008.zip, IndexAccessor.java, IndexAccessor.zip, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[
https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546592
]
Shai Erera commented on LUCENE-1026:
Hey Mark, few more questions:
1. Why is StopWatch needed? it seems like the only thing it does is count the
time of execution of the code. That can be easily achieved by writing this:
long time = System.currentTimeMillis();
code
write any code here
/code
time = System.currentTimeMillis() - time;
System.out.println('time: + time + ms);
Am I missing something?
2. In DefaultIndexAccessor there's this code:
public int writersOut() {
int writers;
synchronized (this) {
writers = this.writerUseCount;
}
return writers;
}
I think it should be replaced with:
public int writersOut() {
return writerUseCount;
}
No need to synchronize on *this* since you're only returning the value, not
performing any modifications to it. The same goes for writingReadersOut,
activeSearchers, isOpen.
What do you think?
3. open() -- is there a real need to throw an exception if someone called the
accessor.open() twice?
I'll attach the modified test and DefaultIndexAccessor files for your review.
I'd apprecite if you can answer the questions.
Provide a simple way to concurrently access a Lucene index from multiple
threads
Key: LUCENE-1026
URL: https://issues.apache.org/jira/browse/LUCENE-1026
Project: Lucene - Java
Issue Type: New Feature
Components: Index, Search
Reporter: Mark Miller
Priority: Minor
Attachments: DefaultIndexAccessor.java,
DefaultMultiIndexAccessor.java, IndexAccessor.java,
IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor.zip,
SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java
For building interactive indexes accessed through a network/internet
(multiple threads).
This builds upon the LuceneIndexAccessor patch. That patch was not very
newbie friendly and did not properly handle MultiSearchers (or at the least
made it easy to get into trouble).
This patch simplifies things and provides out of the box support for sharing
the IndexAccessors across threads. There is also a simple test class and
example SearchServer to get you started.
Future revisions will be zipped.
Works pretty solid as is, but could use the ability to warm new Searchers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546694 ] Shai Erera commented on LUCENE-1026: Hey Mark, I've cleaned up the code, added javadoc and organized the tests. Can you please review the changes - I think the code is even more simpler now. 1. I think we should remove StopWatch. Unless I'm missing something, it looks like it's doing nothing important, inefficiently :-). 2. What is the SimpleSearchServer? Is this like a sample program? Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546702 ] Mark Miller commented on LUCENE-1026: - Hey Mark, few more questions: 1. Why is StopWatch needed? StopWatch could certainly be taken out. I simply used it for timing...to show someone else that it doesn't take that long to get an accessor. I have no problem with removing it. 2. No need to synchronize on this since you're only returning the value, not performing any modifications to it. Its possible this could be taken out. I doubt there would be any noticeable trouble if you tried to read that value while another thread was updating it. 3. open() - is there a real need to throw an exception if someone called the accessor.open() twice? No real need I suppose. I believe this came in from the original code. I have no objections to taking it out. SimpleSearchServer is an attempt to give an example that shows how this code can be used to create a very simple class that allows multi-threaded access to a Lucene index. I whipped it out extremely quickly, but its something I would like to build on. It can be nice to hide the IndexAccessor code behind a simple to use class that hides Lucene and a lot of complexity. Thanks for your work on this! - Mark Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546719 ] Mark Miller commented on LUCENE-1026: - Quick comment: You have the MultiSearcher closing its own IndexAccessor's now. Maybe I messed up copying in your changes, but this causes an Exception in the tests as the IndexFactory also closes all of the IndexAccessor's. (Really, it owns them) It's probably a good idea for the MultiSearcher to have a close (clearing the two caches is not a bad idea), but I don't think the MultiSearcher should close IndexAccessors. It does not own those Accessors and other threads may be using them from the Factory. They are only kept around in the MultiSearcher so that it can release them later, back to the Factory. Also, probably the Factory.close method should call the MultiSearcher close rather than having to get the MuliSearcher and call close separately? Thoughts? - Mark Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
Part of the changes I've done is to have close() not throw exception when it is being called twice (I don't think it's bad to call it twice). Maybe you didn't copy that part of the code? Because the tests run on my machine ... I agree on the last statement. I just didn't want to introduce too many changes at once. But having factory.close() close the MultiSearchAccessor as well is good. If you change that, then the MultiSearcher test can be changed as well to not call close(). You are right about the comment on MultiIndexAccessor's close implementation (not closing its IndexAccessor instances). I must say that its functionality is not very clear, because of lack of documentation (that's why I didn't complete the documentation for this class). Can you update the javadocs while you're changing that? Thanks, Shai. On Nov 29, 2007 3:25 PM, Mark Miller (JIRA) [EMAIL PROTECTED] wrote: [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546719] Mark Miller commented on LUCENE-1026: - Quick comment: You have the MultiSearcher closing its own IndexAccessor's now. Maybe I messed up copying in your changes, but this causes an Exception in the tests as the IndexFactory also closes all of the IndexAccessor's. (Really, it owns them) It's probably a good idea for the MultiSearcher to have a close (clearing the two caches is not a bad idea), but I don't think the MultiSearcher should close IndexAccessors. It does not own those Accessors and other threads may be using them from the Factory. They are only kept around in the MultiSearcher so that it can release them later, back to the Factory. Also, probably the Factory.close method should call the MultiSearcher close rather than having to get the MuliSearcher and call close separately? Thoughts? - Mark Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Regards, Shai Erera
Re: [jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
FYI, any time one thread reads a value while another thread updates it it needs to be synchronized, or with current JVMs a volatile variable. The Java Memory Model requires this. Otherwise you can get a partial value (when the underlying value requires more than one memory access to retrieve). On Nov 29, 2007, at 6:43 AM, Mark Miller (JIRA) wrote: [ https://issues.apache.org/jira/browse/LUCENE-1026? page=com.atlassian.jira.plugin.system.issuetabpanels:comment- tabpanel#action_12546702 ] Mark Miller commented on LUCENE-1026: - Hey Mark, few more questions: 1. Why is StopWatch needed? StopWatch could certainly be taken out. I simply used it for timing...to show someone else that it doesn't take that long to get an accessor. I have no problem with removing it. 2. No need to synchronize on this since you're only returning the value, not performing any modifications to it. Its possible this could be taken out. I doubt there would be any noticeable trouble if you tried to read that value while another thread was updating it. 3. open() - is there a real need to throw an exception if someone called the accessor.open() twice? No real need I suppose. I believe this came in from the original code. I have no objections to taking it out. SimpleSearchServer is an attempt to give an example that shows how this code can be used to create a very simple class that allows multi-threaded access to a Lucene index. I whipped it out extremely quickly, but its something I would like to build on. It can be nice to hide the IndexAccessor code behind a simple to use class that hides Lucene and a lot of complexity. Thanks for your work on this! - Mark Provide a simple way to concurrently access a Lucene index from multiple threads - --- Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/ LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai- IndexAccessor-2.zip, shai-IndexAccessor.zip, shai- IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/ internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
Thanks Robert. Ill keep the sync then. I only considered it possible as the read is for reporting type purposes and so is not relied on for functionality. Sounds like we better retain the sync anyway though. Shai: I have incorporated your code into mine. Looks great so far. There is a rather large MultiIndexAccessor issue that I have to deal with though. It may turn out that MultiSearchers may not be able to be cached. Without passing information about Writer releases to the MultiIndexAccessor, the MultiIndexAccessor will hold onto old Searchers that it shouldn't (the cache never gets cleared). The simple solution is to take out the MultiSearcher caching and just make a new one on every request. This is not a huge deal as the underlying Searchers will be cached, but I liked avoiding the calls to getIndexAcessor if possible. Its prob best to drop the cache though. I think if this is done, your close method on MultiIndexAccessor can be removed. I have added to the tests to expose this issue. I will post updated code later. - Mark robert engels wrote: FYI, any time one thread reads a value while another thread updates it it needs to be synchronized, or with current JVMs a volatile variable. The Java Memory Model requires this. Otherwise you can get a partial value (when the underlying value requires more than one memory access to retrieve). On Nov 29, 2007, at 6:43 AM, Mark Miller (JIRA) wrote: [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546702 ] Mark Miller commented on LUCENE-1026: - Hey Mark, few more questions: 1. Why is StopWatch needed? StopWatch could certainly be taken out. I simply used it for timing...to show someone else that it doesn't take that long to get an accessor. I have no problem with removing it. 2. No need to synchronize on this since you're only returning the value, not performing any modifications to it. Its possible this could be taken out. I doubt there would be any noticeable trouble if you tried to read that value while another thread was updating it. 3. open() - is there a real need to throw an exception if someone called the accessor.open() twice? No real need I suppose. I believe this came in from the original code. I have no objections to taking it out. SimpleSearchServer is an attempt to give an example that shows how this code can be used to create a very simple class that allows multi-threaded access to a Lucene index. I whipped it out extremely quickly, but its something I would like to build on. It can be nice to hide the IndexAccessor code behind a simple to use class that hides Lucene and a lot of complexity. Thanks for your work on this! - Mark Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
I've been thinking about this and I think the situation we are in is okay. The variable is an int and so should be read in one memory access (i would think?). So at worst, the result might be stale? Since this read is just for informational type purposes, (eg its ok if this particular method returns a stale value, by the time you read it its likely to have changed anyway), I think its okay. I think. Everything I read about the Java memory model, I quickly forget. robert engels wrote: FYI, any time one thread reads a value while another thread updates it it needs to be synchronized, or with current JVMs a volatile variable. The Java Memory Model requires this. Otherwise you can get a partial value (when the underlying value requires more than one memory access to retrieve). On Nov 29, 2007, at 6:43 AM, Mark Miller (JIRA) wrote: [ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546702 ] Mark Miller commented on LUCENE-1026: - Hey Mark, few more questions: 1. Why is StopWatch needed? StopWatch could certainly be taken out. I simply used it for timing...to show someone else that it doesn't take that long to get an accessor. I have no problem with removing it. 2. No need to synchronize on this since you're only returning the value, not performing any modifications to it. Its possible this could be taken out. I doubt there would be any noticeable trouble if you tried to read that value while another thread was updating it. 3. open() - is there a real need to throw an exception if someone called the accessor.open() twice? No real need I suppose. I believe this came in from the original code. I have no objections to taking it out. SimpleSearchServer is an attempt to give an example that shows how this code can be used to create a very simple class that allows multi-threaded access to a Lucene index. I whipped it out extremely quickly, but its something I would like to build on. It can be nice to hide the IndexAccessor code behind a simple to use class that hides Lucene and a lot of complexity. Thanks for your work on this! - Mark Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor-2.zip, shai-IndexAccessor.zip, shai-IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
As long as the value displayed is allowed to be wrong or inconsistent you're fine :) The 'fits in int' might hold true but doesn't have to - at a low level in the memory controller it might only write 8 bits, (or 1 bit) at a time. Cache consistency doesn't work as most people think it should. Most of this is very architecture/hardware specific. The value may be VERY STALE, but in practice, when making the cache consistent across multiple processors it is usually done in bulk, so any sync commands cause the entire cache to become up to date. This will change in the future as processor pipelines and OS/JVM support can be more fine grained. Some good overviews on the subject: http://www.cs.umd.edu/~pugh/java/memoryModel/jsr-133-faq.html http://gee.oswego.edu/dl/cpj/jmm.html and the more complete reference... http://www.cs.umd.edu/~pugh/java/memoryModel/ Bottom line, since the value is not used in any computations, you're probably ok. On Nov 29, 2007, at 1:41 PM, Mark Miller wrote: I've been thinking about this and I think the situation we are in is okay. The variable is an int and so should be read in one memory access (i would think?). So at worst, the result might be stale? Since this read is just for informational type purposes, (eg its ok if this particular method returns a stale value, by the time you read it its likely to have changed anyway), I think its okay. I think. Everything I read about the Java memory model, I quickly forget. robert engels wrote: FYI, any time one thread reads a value while another thread updates it it needs to be synchronized, or with current JVMs a volatile variable. The Java Memory Model requires this. Otherwise you can get a partial value (when the underlying value requires more than one memory access to retrieve). On Nov 29, 2007, at 6:43 AM, Mark Miller (JIRA) wrote: [ https://issues.apache.org/jira/browse/LUCENE-1026? page=com.atlassian.jira.plugin.system.issuetabpanels:comment- tabpanel#action_12546702 ] Mark Miller commented on LUCENE-1026: - Hey Mark, few more questions: 1. Why is StopWatch needed? StopWatch could certainly be taken out. I simply used it for timing...to show someone else that it doesn't take that long to get an accessor. I have no problem with removing it. 2. No need to synchronize on this since you're only returning the value, not performing any modifications to it. Its possible this could be taken out. I doubt there would be any noticeable trouble if you tried to read that value while another thread was updating it. 3. open() - is there a real need to throw an exception if someone called the accessor.open() twice? No real need I suppose. I believe this came in from the original code. I have no objections to taking it out. SimpleSearchServer is an attempt to give an example that shows how this code can be used to create a very simple class that allows multi-threaded access to a Lucene index. I whipped it out extremely quickly, but its something I would like to build on. It can be nice to hide the IndexAccessor code behind a simple to use class that hides Lucene and a lot of complexity. Thanks for your work on this! - Mark Provide a simple way to concurrently access a Lucene index from multiple threads --- - Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/ LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai- IndexAccessor-2.zip, shai-IndexAccessor.zip, shai- IndexAccessor3.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/ internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail:
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546288 ] Shai Erera commented on LUCENE-1026: Hi I've downloaded the code and tried to run the tests, but I think there are some problems: 1. The delete() method in the test attempts to delete the directory, and not the underlying files. So in effect it does not do anything. 2. Some of the tests that start new threads don't wait for them (by calling join()). That of course causes some Accessors to be removed (after you call closeAllAccessors()), while those threads are sill running. I've fixed those issues in the test. I'd appreciate if you can take a look. Also, in IndexAccessorFactory I've found some issues: 1. I guess you wanted to have it as a Singleton - so I defined a private default constructor to prevent applications from instantiating it. 2. I modified the code of createAccessor to first lookup if an accessor for that directory already exists. It should save the allocation of DefaultIndexAccessor. 3. I modified the implementation of other methods to access the HashMap of accessors more efficiently. I'd appreciate if you can review my fixes. I'll attach them separately. Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Commented: (LUCENE-1026) Provide a simple way to concurrently access a Lucene index from multiple threads
[ https://issues.apache.org/jira/browse/LUCENE-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12546304 ] Mark Miller commented on LUCENE-1026: - Hey Shai, These fixes are great and I will incorporate them all. I worked this up very quickly based on other less general code I am using. While I have not yet used this code for a project, it will be the framework that I migrate to for future projects. It should see much more development then. I am very eager to add some Searcher warming for one. Also, the tests where whipped together quite quickly. I appreciate your efforts at cleaning them up. Buffing up the SearchServer code to production level will also be on my list. Thanks for your improvements -- if you do any more work, keep me posted. - Mark Provide a simple way to concurrently access a Lucene index from multiple threads Key: LUCENE-1026 URL: https://issues.apache.org/jira/browse/LUCENE-1026 Project: Lucene - Java Issue Type: New Feature Components: Index, Search Reporter: Mark Miller Priority: Minor Attachments: DefaultIndexAccessor.java, DefaultMultiIndexAccessor.java, IndexAccessor.java, IndexAccessorFactory.java, MultiIndexAccessor.java, shai-IndexAccessor.zip, SimpleSearchServer.java, StopWatch.java, TestIndexAccessor.java For building interactive indexes accessed through a network/internet (multiple threads). This builds upon the LuceneIndexAccessor patch. That patch was not very newbie friendly and did not properly handle MultiSearchers (or at the least made it easy to get into trouble). This patch simplifies things and provides out of the box support for sharing the IndexAccessors across threads. There is also a simple test class and example SearchServer to get you started. Future revisions will be zipped. Works pretty solid as is, but could use the ability to warm new Searchers. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
