Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin
The change to make commit consistent is possible during the call-back, I was trying to avoid that level of lock if possible but if it is the contract of that callback then we have to maintain it. Would be nice to be able to do the snapshot first, then release the lock and call the callback on the snapshot instead of holding a lock. But it really isn't built that way currently with only one callback event. Mike Klaas wrote: Hi Jayson, Thanks, I'll take a look in the next few days. The current patch doesn't guarantee index consistency during post-commit callback hooks, right? This could be a problem for index replication. (Incidentally, I'm rather unfamiliar with the new java-based replication design. Anyone care to comment on the implications?) cheers, -MIke On 10-May-09, at 10:54 AM, jayson.minard wrote: Mike, I revamped the DirectUpdateHandler2 into DirectUpdateHandler3 in SOLR-1155, probably ready enough for your review to see if locking makes sense for current Lucene behavior. https://issues.apache.org/jira/browse/SOLR-1155 --j Mike Klaas wrote: On 7-May-09, at 10:36 AM, jayson.minard wrote: Does every thread really need to notify the update handler of the commit interval/threshold being reached, or really just the first thread that notices should send the signal, or better yet a background commit watching thread so that no foreground thread has to pay attention at all. That is assuming they wouldn't need to block like they are now for a reason I'm likely unaware of... This is due to the way Lucene was designed (although recent improvements in Lucene mean we can do better here). See the recent thread Autocommit blocking adds? on solr-user for a related discussion. As the person who first wrote the multi-threaded-ness of DUH2, I'd be very happy to promptly review any improvements made to it. -Mike -- View this message in context: http://www.nabble.com/DirectUpdateHandler2-threads-pile-up-behind-scheduleCommitWithin-tp23431691p23472391.html Sent from the Solr - Dev mailing list archive at Nabble.com. -- View this message in context: http://www.nabble.com/DirectUpdateHandler2-threads-pile-up-behind-scheduleCommitWithin-tp23431691p23502368.html Sent from the Solr - Dev mailing list archive at Nabble.com.
Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin
On Tue, May 12, 2009 at 5:49 AM, Mike Klaas mike.kl...@gmail.com wrote: Thanks, I'll take a look in the next few days. The current patch doesn't guarantee index consistency during post-commit callback hooks, right? This could be a problem for index replication. (Incidentally, I'm rather unfamiliar with the new java-based replication design. Anyone care to comment on the implications?) I have not looked at the patch yet. If adds/deletes are allowed to go through during post-commit callbacks, then it is not a problem for the java based replication. The java based replication uses IndexDeletionPolicy and gets a call back after commits from the IW. Then the last commit point is replicated to the slaves. Even if a backround merge is triggered, the deletion policy ensures that the files of the commit point being replicated are preserved until the replication finishes. -- Regards, Shalin Shekhar Mangar.
Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin
Hi Jayson, Thanks, I'll take a look in the next few days. The current patch doesn't guarantee index consistency during post-commit callback hooks, right? This could be a problem for index replication. (Incidentally, I'm rather unfamiliar with the new java-based replication design. Anyone care to comment on the implications?) cheers, -MIke On 10-May-09, at 10:54 AM, jayson.minard wrote: Mike, I revamped the DirectUpdateHandler2 into DirectUpdateHandler3 in SOLR-1155, probably ready enough for your review to see if locking makes sense for current Lucene behavior. https://issues.apache.org/jira/browse/SOLR-1155 --j Mike Klaas wrote: On 7-May-09, at 10:36 AM, jayson.minard wrote: Does every thread really need to notify the update handler of the commit interval/threshold being reached, or really just the first thread that notices should send the signal, or better yet a background commit watching thread so that no foreground thread has to pay attention at all. That is assuming they wouldn't need to block like they are now for a reason I'm likely unaware of... This is due to the way Lucene was designed (although recent improvements in Lucene mean we can do better here). See the recent thread Autocommit blocking adds? on solr-user for a related discussion. As the person who first wrote the multi-threaded-ness of DUH2, I'd be very happy to promptly review any improvements made to it. -Mike -- View this message in context: http://www.nabble.com/DirectUpdateHandler2-threads-pile-up-behind-scheduleCommitWithin-tp23431691p23472391.html Sent from the Solr - Dev mailing list archive at Nabble.com.
Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin
On 7-May-09, at 10:36 AM, jayson.minard wrote: Does every thread really need to notify the update handler of the commit interval/threshold being reached, or really just the first thread that notices should send the signal, or better yet a background commit watching thread so that no foreground thread has to pay attention at all. That is assuming they wouldn't need to block like they are now for a reason I'm likely unaware of... This is due to the way Lucene was designed (although recent improvements in Lucene mean we can do better here). See the recent thread Autocommit blocking adds? on solr-user for a related discussion. As the person who first wrote the multi-threaded-ness of DUH2, I'd be very happy to promptly review any improvements made to it. -Mike
Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin
Does every thread really need to notify the update handler of the commit interval/threshold being reached, or really just the first thread that notices should send the signal, or better yet a background commit watching thread so that no foreground thread has to pay attention at all. That is assuming they wouldn't need to block like they are now for a reason I'm likely unaware of... --j -- View this message in context: http://www.nabble.com/DirectUpdateHandler2-threads-pile-up-behind-scheduleCommitWithin-tp23431691p23431745.html Sent from the Solr - Dev mailing list archive at Nabble.com.
