[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13057833#comment-13057833 ] Mark Miller commented on SOLR-2193: --- This issue is superceded by: SOLR-2565 Prevent IW#close and cut over to IW#commit Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13057834#comment-13057834 ] Mark Miller commented on SOLR-2193: --- bq. Curious; why is the resolution status invalid? Dunno - it's not invalid. I've re-resolved as duplicate Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053678#comment-13053678 ] Simon Willnauer commented on SOLR-2193: --- bq. Curious; why is the resolution status invalid? well we decided to cut this into two new issues and close this one. see: bq.further developments in SOLR-2565 and SOLR-2566 there have been discussions about the focus here so we made it more clear. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Robert Muir Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041995#comment-13041995 ] Jason Rutherglen commented on SOLR-2193: I'm curious if someone who doesn't work at Lucid can be involved in Solr design discussions. In any case, please autocratically continue. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Robert Muir Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041997#comment-13041997 ] Robert Muir commented on SOLR-2193: --- I committed this patch (with some additional ref counting checks in SolrTestCaseJ4) to https://svn.apache.org/repos/asf/lucene/dev/branches/solr2193 Jason (or anyone else!): if you want to be involved, please contribute constructively in the form of useful ideas and patches against this branch. In the meantime, I will be trying to improve the tests for what exists now, as its the best way (not manual reviews) to increase my confidence. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Robert Muir Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042004#comment-13042004 ] Jason Rutherglen commented on SOLR-2193: This article is an indicator of the types of benchmarks to perform: http://engineering.socialcast.com/2011/05/realtime-search-solr-vs-elasticsearch/ Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Robert Muir Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042009#comment-13042009 ] Robert Muir commented on SOLR-2193: --- Jason, this issue isn't intended to solve NRT. You might want to create a followup issue for improved NRT support. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Robert Muir Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042011#comment-13042011
]
Jason Rutherglen commented on SOLR-2193:
bq. Jason, this issue isn't intended to solve NRT
What is this line doing?
{code}
newReader = currentReader.reopen(indexWriterProvider.getIndexWriter(), true);
{code}
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Assignee: Robert Muir
Fix For: 4.0
Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch,
SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042013#comment-13042013
]
Jason Rutherglen commented on SOLR-2193:
Also:
https://issues.apache.org/jira/browse/SOLR-2193?focusedCommentId=13016875page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13016875
And this comment:
{quote}
Can you elaborate on why you don't think it's implementing NRT? I've tested
basic indexing/searching using wikipedia documents at about 50-100 documents a
second, opening a new reader every second. That felt pretty near-real-time to
me, but the phrase is subjective.
{quote}
from:
https://issues.apache.org/jira/browse/SOLR-2193?focusedCommentId=13041268page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13041268
Robert, your statement's confusing.
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Assignee: Robert Muir
Fix For: 4.0
Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch,
SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042015#comment-13042015 ] Robert Muir commented on SOLR-2193: --- Its not confusing at all... its a step towards NRT, thats all. the main improvement to me, is to not close the indexwriter this patch doesn't need to fully make solr realtime. you can keep whining that it isn't, but this doesn't accomplish anything. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Robert Muir Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042026#comment-13042026 ] Simon Willnauer commented on SOLR-2193: --- I intend to close this issue and go on in SOLR-2565 / SOLR-2566 I hope these issues provide a more clear focus on what the issues trying to address. If nobody objects I am going to close this today. simon Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Robert Muir Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042032#comment-13042032 ] Robert Muir commented on SOLR-2193: --- +1 Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Robert Muir Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042146#comment-13042146 ] Ryan McKinley commented on SOLR-2193: - +1 this is a great step forward. better NRT support can come in later patches. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Robert Muir Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13042259#comment-13042259 ] Jason Rutherglen commented on SOLR-2193: Simon, thanks for opening new issues. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Robert Muir Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041464#comment-13041464 ] Jason Rutherglen commented on SOLR-2193: bq. I enjoyed our dialogue honestly I'd prefer to simply get things done rather than banter with no results. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041547#comment-13041547 ] Mark Miller commented on SOLR-2193: --- Heh - you approached me the wrong way then. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041554#comment-13041554 ] Simon Willnauer commented on SOLR-2193: --- Jason, your action on this issue is not appropriate and with my PMC Hat on I can't tolerate this! I want remind you to move back to an appropriate tone and help out rather than randomly -1. I don't see any political issues here neither is the -1 appropriate. You have done lots of realtime work and mark seems to be pretty open to comments too so get back to technical discussion. If you have any general issues feel free to raise them on dev@l.a.o simon Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041688#comment-13041688 ] Ryan McKinley commented on SOLR-2193: - Reading over the comments here and without knowing the details, it looks like a big improvement over what we currently have. Jason -- is your concern that this is a fundamentally wrong direction, or that it does not go far enough to support NRT? Can this patch be improved? Can it be committed, then improved? Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041691#comment-13041691 ] Jason Rutherglen commented on SOLR-2193: As previously suggested, we need a new issue that refactors IndexWriter into SolrCore, instead of placing it into an UpdateHandler. Then we can iterate on re/factoring the NRT functionality. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041693#comment-13041693
]
Jason Rutherglen commented on SOLR-2193:
{quote}this is a fundamentally wrong direction{quote}
Yes. The idea of adding NRT is good though.
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Fix For: 4.0
Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch,
SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [jira] [Commented] (SOLR-2193) Re-architect Update Handler
Taking some of Jason's concerns here (to the dev list)... we should stick to technical feedback on the issue: On Mon, May 30, 2011 at 11:54 PM, Jason Rutherglen (JIRA) j...@apache.org wrote: It's been clear for quite a while that you folks at Lucid are trying to protect your golden goose, eg, Solr from changing much unless dictated by your staff or a paying customer. I think in politics those are called bribes? While Apache must always be vigilant to ensure that no commercial entity, neither Lucid nor for example my sponsor (IBM), abuses its influence over any project, I don't see any evidence that Lucid is doing so here. If Lucid did somehow conspire to fight improvements to Solr, that would be severely detrimental to its own future. Apache very much needs corporations to have a stake in projects, so that these corporations sponsor individual's time/itch towards improving things. As long as the needs of that sponsor generally align well with the needs of the project, as is happening here in my opinion, it's win/win. It's also a matter of personal integrity: if there is a severe conflict of interest, where the sponsor wants something changed but the committer realizes it goes against the project / the Apache way / etc., then the committer should simply say no to the sponsor. If that becomes a pattern, the committer should move on to a different sponsor. I know the active Lucene/Solr committers well enough to know that they all possess this integrity, so I don't see any way Lucid could conspire here. Hence a large part of the recent fracas regarding modularizing the goose, whose 'resolution' has resulted in no changes. I don't think Lucid was somehow conspiring here; I think certain individuals simply had differences in opinion. This is par for the course in open source ;) If two people always agree, one is not needed... disagreement is healthy. And I disagree that there have been no changes; in fact, it's the reverse: the conclusion/consensus on that refactoring thread is that people who have the itch to refactor/poach are entirely free to do so, as long as it does not harm Lucene/Solr. Refactoring/poaching is the bread and butter of open source, one of our basic rights. The refactoring of the analysis module is a great example. More recently, the grouping module, and then the sudden fast iterations adding strong improvements to its capabilities, soon to bring grouping to 3.x Solr, is another. The suggest module was factored out; function queries has an initial baby step patch. There's an issue opened to factor out faceting. Mike McCandless http://blog.mikemccandless.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [jira] [Commented] (SOLR-2193) Re-architect Update Handler
If Lucid did somehow conspire to fight improvements to Solr, that would be severely detrimental to its own future. The arguments were brought up previously regarding how in the name of stability refactoring Solr does not occur. It's a touchy subject that yes can be technically justified, however in most cases comes from Lucid staff. This leads one on a rather short line to the conclusion that they don't want the Goose tampered with. Many times it seems like the Spruce Goose however. On Tue, May 31, 2011 at 10:40 AM, Michael McCandless luc...@mikemccandless.com wrote: Taking some of Jason's concerns here (to the dev list)... we should stick to technical feedback on the issue: On Mon, May 30, 2011 at 11:54 PM, Jason Rutherglen (JIRA) j...@apache.org wrote: It's been clear for quite a while that you folks at Lucid are trying to protect your golden goose, eg, Solr from changing much unless dictated by your staff or a paying customer. I think in politics those are called bribes? While Apache must always be vigilant to ensure that no commercial entity, neither Lucid nor for example my sponsor (IBM), abuses its influence over any project, I don't see any evidence that Lucid is doing so here. If Lucid did somehow conspire to fight improvements to Solr, that would be severely detrimental to its own future. Apache very much needs corporations to have a stake in projects, so that these corporations sponsor individual's time/itch towards improving things. As long as the needs of that sponsor generally align well with the needs of the project, as is happening here in my opinion, it's win/win. It's also a matter of personal integrity: if there is a severe conflict of interest, where the sponsor wants something changed but the committer realizes it goes against the project / the Apache way / etc., then the committer should simply say no to the sponsor. If that becomes a pattern, the committer should move on to a different sponsor. I know the active Lucene/Solr committers well enough to know that they all possess this integrity, so I don't see any way Lucid could conspire here. Hence a large part of the recent fracas regarding modularizing the goose, whose 'resolution' has resulted in no changes. I don't think Lucid was somehow conspiring here; I think certain individuals simply had differences in opinion. This is par for the course in open source ;) If two people always agree, one is not needed... disagreement is healthy. And I disagree that there have been no changes; in fact, it's the reverse: the conclusion/consensus on that refactoring thread is that people who have the itch to refactor/poach are entirely free to do so, as long as it does not harm Lucene/Solr. Refactoring/poaching is the bread and butter of open source, one of our basic rights. The refactoring of the analysis module is a great example. More recently, the grouping module, and then the sudden fast iterations adding strong improvements to its capabilities, soon to bring grouping to 3.x Solr, is another. The suggest module was factored out; function queries has an initial baby step patch. There's an issue opened to factor out faceting. Mike McCandless http://blog.mikemccandless.com - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041711#comment-13041711 ] Michael McCandless commented on SOLR-2193: -- I haven't looked that closely a this patch yet, but it already fixes a long standing problem in Solr, that a long running merge blocks a Solr commit, because it switches to IW.commit instead of closing/opening the writer. This seems like a great step forward? Sure, we should pursue moving IW to SolrCore, cutting over to IW's NRT's APIs, making all caching per-segment, etc., but these should not block committing this improvement; they can be pursued as followon/parallel issues: progress not perfection. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041715#comment-13041715
]
Jason Rutherglen commented on SOLR-2193:
{quote}I haven't looked that closely a this patch yet, but it already fixes a
long standing problem in Solr, that a long running merge blocks a Solr commit,
because it switches to IW.commit instead of closing/opening the writer.{quote}
Yes, that is/was not clear in the issue. Thank you for spelling it out.
However I think the patch is creating new abstract classes, that would then go
away? Why not spend a little more time trying to do a more overall design for
future refactoring?
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Fix For: 4.0
Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch,
SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041749#comment-13041749
]
Robert Muir commented on SOLR-2193:
---
{quote} However I think the patch is creating new abstract classes, that would
then go away? {quote}
Just one interface right, IndexWriterProvider?
Mark seems to describe the reasoning behind this pretty well:
{quote}
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
I've made an attempt at this with the IndexWriterProvider and some simple ref
counting that could use a once over. This changes how you should reload a
Solrcore - rather than just opening a new one and closing the old one, you want
to call a method on the SolrCore itself and get the new SolrCore.
{quote}
and in the original goals
{quote}
solving our rather nasty reload a core, briefly different writers on the same
index problem (usually avoided because the overlap is brief and the IndexWriter
created lazily).
{quote}
I'm at best a Solr dummy, but it seems to me to be a good tradeoff. We could
toss @lucene.internal or @lucene.experimental on this so its clear that things
are still in flux?
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Fix For: 4.0
Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch,
SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041767#comment-13041767
]
Jason Rutherglen commented on SOLR-2193:
{quote}solving our rather nasty reload a core, briefly different writers on the
same index problem (usually avoided because the overlap is brief and the
IndexWriter created lazily).{quote}
Robert I fully agree, however then the title of the Jira is incorrect.
Also the whole ref counted thing in Solr:
{code}
RefCountedSolrIndexSearcher holder = core.getNewestSearcher(false);
SolrIndexSearcher s = holder.get();
holder.decref();
// since there could be two commits in a row, don't test for a specific new
searcher
// just test that the old one has been replaced.
{code}
Should not be needed anymore. We're also adding ref counting on IWs now as
well? All of this is unnecessary. If we're modularizing, this isn't right
path to go one.
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Fix For: 4.0
Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch,
SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041771#comment-13041771 ] Ryan McKinley commented on SOLR-2193: - bq. we need a new issue that refactors IndexWriter into SolrCore, instead of placing it into an UpdateHandler Can't that be done as an iteration/improvement? Nothing here blocks that possible change -- and this patch sorts out a bunch of stuff that needs to be taken care of anyway. Recent dev on /trunk has been fast and iterative (very exciting!) -- this patch seems like a good way to get the ball rolling and move forward. Lets mark anything as @lucene.internal that has viable alternatives and start smaller patches to move that along. I apologize that this stuff is out of *my* league, but the smaller the issue the easier it is to understand and actually get committed. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041776#comment-13041776 ] Robert Muir commented on SOLR-2193: --- bq. Robert I fully agree, however then the title of the Jira is incorrect. How is it incorrect? The description of the issue is to overhaul the UpdateHandler, not to rearchitect Solr for NRT, move IW into SolrCore, or other things. bq. We're also adding ref counting on IWs now as well? All of this is unnecessary. If we're modularizing, this isn't right path to go one. How do you propose supporting core reloads without it? It seems to me (again solr dummy) that some mechanism like this is needed. I guess personally, just looking at the big picture, this issue seems like a great win for Solr. I tested the patch and all tests pass... if there are technical concerns with test coverage I'm happy to try to help there to help see this thing committed. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041974#comment-13041974 ] Shai Erera commented on SOLR-2193: -- bq. The concept of Solr continuing to be difficult to customize is not so great. I haven't looked at the patch at all, as I'm not familiar w/ Solr code and cannot comment on the technical issues here. But I do would like to understand if this patch indeed gets us further from being able to customize Solr in the future or not. By customizing I mean (and I realize not all of it is handled in this issue): # Control the IW instance through not only properties, but IW extensions (for e.g. overriding addDocument / commit so that other custom data structures can be updated at the same time as well). # Control the replication process, for e.g., replicating additional files, not only IW files, that are required by the runtime logic. # Control the IndexSearcher instance, or the query execution engine, in order to run queries at the low-level. i.e., being able to call IS.search(Q, Collector), and not from some high-level API where I can only submit a String and get back Result. These are important extension points, IMO, to Solr that we can offer applications. I admit, really advanced applications that know what they do and want, that just don't want to rewrite Solr entirely in order to execute queries differently. NOTE: I am not trying, nor do I want, to discuss here whether it should be the roadmap of Solr (i.e., provide all sorts of extensions that enable the above), but merely asking if that patch indeed gets us further from accomplishing that. I will not cast any +/-1 vote if it does, just (being a Solr dummy, not even at best) trying to understand if that's really what's being proposed here. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041986#comment-13041986
]
Robert Muir commented on SOLR-2193:
---
{quote}
-1 on the implementation which introduces even more awkward layers into Solr,
which should be going in the direction of removing the old cruft.
{quote}
Sorry Jason, you cannot simply say -1 without answering my questions about how
these problems can be solved in an alternative way.
I hate to be an asshole, but your vote is non-binding. I'm going to take this
issue to the finish line now.
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Assignee: Robert Muir
Fix For: 4.0
Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch,
SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041235#comment-13041235 ] Mark Miller commented on SOLR-2193: --- There are a few things I'd like to do first, but I think we should likely commit this sooner rather than later, then we can iterate on trunk. I don't think I'll get to the few things I want to before buzzwords (and then I'm taking a week and a half semi vaca after that), but I plan to push for commit after that if I've hit anything I consider a blocker. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041261#comment-13041261
]
Mark Miller commented on SOLR-2193:
---
Reporting the current state regarding original goals:
bq. 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
I want to do this, but it's just easier to handle after the first commit.
{quote} 2. Expose the SolrIndexWriter in the api or add the proper abstractions
to get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish {quote}
There is no need for this anymore.
bq. 3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
This is done.
bq. 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
This is done.
bq. 5. Keep NRT support in mind.
Always ;) This patch certainly won't complete the NRT work needed, but it's a
large start, and a huge step forward for Solr. Much of what is left to do is in
this area though.
bq. 6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
Have not really addressed anything here - don't think I hurt anything either
though. Worth considering the new IndexWriterProvider and how it might relate
to this though.
{quote}7. Address the current issues we face because multiple
original/'reloaded' cores can have a different IndexWriter on the same
index.{quote}
I've made an attempt at this with the IndexWriterProvider and some simple ref
counting that could use a once over. This changes how you should reload a
Solrcore - rather than just opening a new one and closing the old one, you want
to call a method on the SolrCore itself and get the new SolrCore.
More to come. I've updated the patch to trunk as well - will attach shortly.
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Assignee: Mark Miller
Fix For: 4.0
Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch,
SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041265#comment-13041265 ] Jason Rutherglen commented on SOLR-2193: I think the Solr ref counting code should go/exit, it's prone to pile up. Instead as with Twitter's system, a new reader is opened per query, because the readers are lightweight enough. I think that's a better path to pursue than monkey wrenching Solr's existing system which from the ground up, is not designed for NRT. If this patch isn't implementing NRT, what is the point? Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041268#comment-13041268 ] Mark Miller commented on SOLR-2193: --- Hi Jason - long time no see :) bq. I think the Solr ref counting code should go/exit, it's prone to pile up. Can you elaborate? I'm def open to other approaches. I'm not yet sure why you are concerned about pile up? Whenever a new SolrCore is opening, the previous ref will be closing. Have you tried the patch and seen a problem here? bq. Instead as with Twitter's system, a new reader is opened per query, because the readers are lightweight enough. Hmmm...opening a new IndexReader is not usually lightweight at all right now...which is why we use reopen. bq. If this patch isn't implementing NRT, what is the point? Can you elaborate on why you don't think it's implementing NRT? I've tested basic indexing/searching using wikipedia documents at about 50-100 documents a second, opening a new reader every second. That felt pretty near-real-time to me, but the phrase is subjective. There are remaining issues - Solr does not currently do everything at a per segment level as you know - but per segment use cases will be very fast, and non per segment use cases will be *much* more NRT too. You are questioning my whole patch :) Which I am happy about - but I'm wondering if you have actually reviewed it - if not, I wish you would! Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041275#comment-13041275 ] Mark Miller commented on SOLR-2193: --- Also - if it helps clarify: This issue simply starts taking advantage of Lucene's NRT work. Much of that is being improved as some of the cool ideas from the Twitter work are being made more generally applicable and integrated into trunk Lucene. This issue does not address NRT at that low level - for that see the Lucene JIRA issues on this topic. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041276#comment-13041276 ] Jason Rutherglen commented on SOLR-2193: bq. This patch certainly won't complete the NRT work needed Mark, I was reading this comment. bq. You are questioning my whole patch I think it'll be easier to add what's needed for this patch into Lucene rather than retrofit Solr. I mentioned this a while back however there was pushback on re-architecting Solr. Making everything per-segment would be much more productive than allowing NRT at this stage. Ah, I think you're simply trying to avoid the stop the world Solr has right now? If so that should be more prevalent in the Jira. bq. IndexWriter writer = ((DirectUpdateHandler2)core.getUpdateHandler()).getIndexWriterProvider().getIndexWriter(); Ugly Solr style code?! The commit in X time can be simple contrib class for Lucene. It doesn't need to be Solr specific. Anyways I tried to do this 2 years ago for NRT, there was pushback just get the IndexWriter like the above code from the update handler. politicalWow/political Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041277#comment-13041277 ] Mark Miller commented on SOLR-2193: --- bq. I think it'll be easier to add what's needed for this patch into Lucene rather than retrofit Solr. This makes no sense :) See my above comment. bq. Ugly Solr style code?! This is simply to detect the case where the update handler was changed on reload - not something we even necessarily have to support - but a case I did consider. If you can suggest better code to handle such a case, please do. bq. The commit in X time can be simple contrib class for Lucene. It doesn't need to be Solr specific. Moving the location of the auto commit code would be a very different issue. Not one that makes sense to me right now either. bq. Anyways I tried to do this 2 years ago for NRT, there was pushback just get the IndexWriter like the above code from the update handler. politicalWow/political I remember that issue - I tried to make some comments to help you out with it - you never managed to get it very far if memory serves though - so now I'm taking my own crack at it, along with addressing a few other pet peeves I have. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041307#comment-13041307 ] Mark Miller commented on SOLR-2193: --- P.S. While it may not be believed, there are no politics behind this issue :) I did not intentionally side step any other issues - nor did Lucid even initiate that I work on this. I wanted to fix the update handler - I didn't want to let any history or pre conceived notions get in my way. Was I successful at that? I don't know. But I approached this as blank slate as I could - if I could change the update handler, what would I do? I didn't look at what anyone else had done. So far, this is it. The community can take my effort or reject it, just like any others out there. The motivation of this issue, as far as I am concerned, is my own self interest in doing some interesting coding, and my good will towards Lucene/Solr. Any other motivations where so minor they don't even register. Take it or leave - I'm not looking to cram this down anyones throat. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041399#comment-13041399 ] Jason Rutherglen commented on SOLR-2193: bq. IndexWriter writer =((DirectUpdateHandler2)core.getUpdateHandler()).getIndexWriterProvider().getIndexWriter(); Why isn't IW a part of SolrCore? It's the main class running the show. How can there be a Solr core without an IW? I think IW never gets closed until the SolrCore is closed. The next move would be to place all of the caches at the segment level. It's been clear for quite a while that you folks at Lucid are trying to protect your golden goose, eg, Solr from changing much unless dictated by your staff or a paying customer. I think in politics those are called bribes? Hence a large part of the recent fracas regarding modularizing the goose, whose 'resolution' has resulted in no changes. It's astonishing the changes that are OK for Solr by some people, that are no OK from others. This is not a meritocracy. If you insist on driving, you should incorporate some of the feedback given. Solr was hacked together from the beginning and this is yet another ugly retrofit that is being steamrolled in. If you're confident in your abilities you're confident enough to make major changes. I've never seen that on the Solr side of the Lucene project. bq. I remember that issue - I tried to make some comments to help you out with it No there was push back on something silly and simple, eg, getting the IW from the UpdateHandler, just as you have done here. What is the point in contributing when they are blocked for no reason? bq. SOLR-1155 What happened to this poor guys patch? Nothing. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041408#comment-13041408 ] Mark Miller commented on SOLR-2193: --- Heh. Golden Goose. There are geese on the farm I grew up next too. They never needed much protected - they ran in a pack and charged around fearless. If I had a Golden Goose (and I'm still looking if you know anyone), I'd choose to protect it with like a knife or a gun rather than code :) This is a meritocracy. Those with merit impart it on the deserving. There are a wide variety of those with merit from a diverse set of companies. So others that deserve merit can be sure to get a fair shake over time. Those without merit...well...they will be what they be. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041414#comment-13041414 ] Jason Rutherglen commented on SOLR-2193: Mark, That's an odd non-technical answer, and in the meritocracy of comedy, not funny either. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041420#comment-13041420
]
Mark Miller commented on SOLR-2193:
---
Plus this is a holiday :( Do I really have to take unfounded lucid crap even in
my free time?
Well, I'm game. I've pulled out my bottle of wine, and I'm ready to answer your
questions:
bq. Why isn't IW a part of SolrCore?...
I leave this as an exercise for the reader.
bq. I think in politics those are called bribes?
I want it always to be known - that I will in fact take bribes. In fact, I'd
like to take my first one any time anyone is willing to pay me. As this is a
legally gray area, you will have to understand that I cannot place any
guarantees on my work. Also, I'm currently only accepting bitcoin.
bq. If you insist on driving, you should incorporate some of the feedback given.
I don't need to drive. I love it when others drives. Some people are such good
drivers! Trust me - I was never one to jump to drive. I can be fairly absent
minded. My friends often preferred to drive. But by golly, there where some
people even I wouldn't drive with...sometimes cause I was never needing to go
anywhere when they where driving, and other times cause they where scary
drivers!
bq. If you're confident in your abilities you're confident enough to make
major changes. I've never seen that on the Solr side of the Lucene project.
I'm sorry! I will try harder! My abilities are not insane. But I think they are
good. I've never been great with self rating, but I think I'm a solid above
average. I'm even better at 4v4 Warcraft3, all archer strategy. If not for
Korea, that is how I'd make my money. It would be way easier to accumulate
bitcoin that way.
bq. Solr was hacked together from the beginning and this is yet another ugly
retrofit that is being steamrolled in.
Steamrolled? You hurt me...I only just above I don't won't to steam roll this
issue that I am working in my free time of my own volition? My you are a fun
guy :)
{quote}
No there was push back on something silly and simple, eg, getting the IW
from the UpdateHandler, just as you have done here. What is the point in
contributing when they are blocked for no reason?
{quote}
Jason, that is in test code :) My approach is plenty different from what you
are up to :)
bq. What happened to this poor guys patch? Nothing.
I had never seen this guys patch. It's not my job to jump on anyones patch if I
did. Since I have started my rearchitect issue, I have seen a couple issues out
there that do a small piece of what I am up to. I have looked at a couple, and
they did not work how I wanted. You can try it a different way. Honestly, if
you write some code that I like, I will commit it for you. But I will only
commit what I'm comfortable with. Lucky for you there are a bunch of us. Many
not from Lucid. All you gotto do is convince one of them to take on your issue
or tackle your code.
Personally, I make my contributions lately out of my own time and will. If it
bothers you so much, I'll be happy to hold off - those that voted for this
issue can wait for someone else to fix this at the lucene level like twitter.
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Assignee: Mark Miller
Fix For: 4.0
Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch,
SOLR-2193.patch, SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041422#comment-13041422 ] Jason Rutherglen commented on SOLR-2193: Mark I think you're missing the point. If you're committer then it's implied you review patches and interact with the community, nicely. That's not happening with in this issue, or in Solr as noted by in fact many people. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041427#comment-13041427 ] Mark Miller commented on SOLR-2193: --- Hey man - I can only do what I can do. Let god sort it out. I'm ready to be judged. I didn't do it all perfect. I didn't make all the right choices. But overall I'd say I was way more good than bad. I may have been a little sociopathic during that one short stint in my teens, but other than that, its worked out well I think. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041428#comment-13041428 ] Jason Rutherglen commented on SOLR-2193: -1 on the patch, I just reviewed again. IndexWriter should be a part of SolrCore (IW is canonical), as we should not be opening and closing IWs in the life of a Solr core. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041431#comment-13041431
]
Chris Male commented on SOLR-2193:
--
Guys, this has gotten beyond ridiculous. Lets stick to the goal of helping
Mark complete this work. If there's concerns about issues being ignored or
whatever, lets address that elsewhere without the accusations and sarcasm.
{quote}
bq. Why isn't IW a part of SolrCore?...
I leave this as an exercise for the reader.
{quote}
I don't know if this is immediately obvious from the code (and consequently
doesn't need explaining), but it seems on the surface to be a reasonable
question to ask. Are you able to say why this isn't a preferable direction to
take? The answer might prove useful to others too.
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Assignee: Mark Miller
Fix For: 4.0
Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch,
SOLR-2193.patch, SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
7. Address the current issues we face because multiple original/'reloaded'
cores can have a different IndexWriter on the same index.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041432#comment-13041432 ] Mark Miller commented on SOLR-2193: --- bq. -1 on the patch, I just reviewed again. IndexWriter should be a part of SolrCore (IW is canonical), as we should not be opening and closing IWs in the life of a Solr core. Okay, -1 accepted. You win, good fight. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041433#comment-13041433 ] Jason Rutherglen commented on SOLR-2193: bq. Okay, -1 accepted. You win, good fight Mark this was no fight, this is the open source Apache way. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13041434#comment-13041434 ] Mark Miller commented on SOLR-2193: --- I meant in the figurative sparring sense, not negative fight sense. Honestly Jason, you made my night :) I enjoyed our dialogue honestly. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13034818#comment-13034818 ] Mark Miller commented on SOLR-2193: --- I've got some fixes for this, and I've started on some tests and other minor steps forward. I'll put it up before too long. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13034918#comment-13034918 ] Mark Miller commented on SOLR-2193: --- Next I need to look at the thread safety of CommitTracker under the new locking system. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13035051#comment-13035051 ] Mark Miller commented on SOLR-2193: --- If we go with this separate softAutoCommit as an option, still need to think about overlapping hard/soft commits. Eg you might want to do a soft commit every 4 seconds and a hardcommit every 16 seconds, but on the 16th second you don't necessarily want to do both types of commit (though not likely that big a deal). I've got logic to avoid this in the commit by doc case, but nothing for the time based auto commit. CommitWithin support is also still an interesting additional option - as well as Yonik's adea about specifying a staleness hint at query time. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13035075#comment-13035075 ] Grant Ingersoll commented on SOLR-2193: --- Crazy idea: drop the notion of commits all together (or make it an expert thing for the hard core). Default it to 1 second. I wonder how all of this plays with warming/caching, etc. Do you even need those things in this type of setup? Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13035112#comment-13035112 ] Hoss Man commented on SOLR-2193: bq. I think that commitWithin should mean soft commit. Users of commitWithin care about when the changes become visible, not when they are guaranteed to be fsync'd. while i don't know that we can assume that's the expectation of *all* existing commitWithin users, i think it probably is safe to assume that the users who *do* expect commitWithin to refer to fsync can be expected to pay attention enough if we add a new hardCommitWithin=35 option and start using that if it's what they want. Alternately: add commitWithin.style=(hard|soft) where the default is soft and let people specify it as a default on their update request handlers if the behavior they really want is hard Or simplify things even further: eliminate softCommit as an explicit type of action and add a new commit.type=(hard|soft) param exists -- commit.type would affect both explicitly requested commits, and commitWithin. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13019953#comment-13019953 ] Jayson Minard commented on SOLR-2193: - Some of this was already solved in: https://issues.apache.org/jira/browse/SOLR-1155 (locking and re-opening index writer were fixed) Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13019961#comment-13019961 ] Jayson Minard commented on SOLR-2193: - Since SOLR-1155 is probably an easier change for Solr 3.1 due to its ancestry, so to get the same benefits I'll work to update it for that version, assuming this patch of yours is for 4.x onwards. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13019983#comment-13019983 ] Mark Miller commented on SOLR-2193: --- Yes - my plan for this was 4.x. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13018105#comment-13018105 ] Mark Miller commented on SOLR-2193: --- Also need to consider how we should work in IndexReaderWarmer. Have not thought about it in depth yet, but it could optionally reuse the warming queries? Or have a separate conf section? Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13018111#comment-13018111 ] Yonik Seeley commented on SOLR-2193: bq. it could optionally reuse the warming queries? Off the top of my head, I don't see a reason to differentiate between hard/soft commits wrt warming in general. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13018113#comment-13018113 ] Mark Miller commented on SOLR-2193: --- Hmm...yeah - doesn't seem like we could get away with only warming the one segment in Solr... Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13017909#comment-13017909 ] Mark Miller commented on SOLR-2193: --- So one issue to consider: If we start sharing the IndexWriter across SolrCores like this, on SolrCore reload, if you have changed IndexWriter settings, they will not take affect. My thinking here is that that is not really a big deal. I'm thinking perhaps we might do the following: 1. On reload, print a log warning that if IndexWriter settings have changed, they have not been picked up? 2. Add another param to the update/commit cmd that lets you force the opening of a new IndexWriter with the latest settings. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13017912#comment-13017912 ] Mark Miller commented on SOLR-2193: --- If we really wanted, we could also apply the 'live' setting changes on the fly rather than opening a new IndexWriter if we wanted to detect that case. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch, SOLR-2193.patch, SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) 7. Address the current issues we face because multiple original/'reloaded' cores can have a different IndexWriter on the same index. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[
https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13017461#comment-13017461
]
Mark Miller commented on SOLR-2193:
---
{quote} I wonder how this should work with autocommit?
Someone may want a soft/reopen autocommit once every x seconds, but still may
want a hard flush to stable storage in case I crash commit at some other
duration. {quote}
Right - I agree. How about another simple start? Simply add another
commitTracker that does soft commits - then you can schedule a mix of soft and
hard commits.
{quote}
The other thing that might be cool is a client-specified freshness per request.
For example, when they pass in a query, they specify that they need data that's
no more than 1 second old... and if it's too old that will trigger a reopen
(and block that specific request until the new searcher can be used). The
benefit here is that big bulk uploads won't be interrupted if there is no time
sensitive query traffic. The downside is that a high latency may be exposed to
those requests if they depend on stuff that can take a lot of time the first
time (like faceting).
{quote}
Yeah - I remember you mentioning this before - I def think this would be cool -
perhaps as a follow on issue - though hopefully the affect on bulk updates will
be minimized when Lucene takes care of the 'flush blocks the world' issue.
Re-architect Update Handler
---
Key: SOLR-2193
URL: https://issues.apache.org/jira/browse/SOLR-2193
Project: Solr
Issue Type: Improvement
Reporter: Mark Miller
Assignee: Mark Miller
Fix For: 4.0
Attachments: SOLR-2193.patch
The update handler needs an overhaul.
A few goals I think we might want to look at:
1. Cleanup - drop DirectUpdateHandler(2) line - move to something like
UpdateHandler, DefaultUpdateHandler
2. Expose the SolrIndexWriter in the api or add the proper abstractions to
get done what we now do with special casing:
if (directupdatehandler2)
success
else
failish
3. Stop closing the IndexWriter and start using commit (still lazy IW init
though).
4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level.
5. Keep NRT support in mind.
6. Keep microsharding in mind (maintain logical index as multiple physical
indexes)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13016862#comment-13016862 ] Mark Miller commented on SOLR-2193: --- I've started experimenting with some simple Lucene near-realtime support, which lets you avoid the commit on reopen as well. Even if everything in Solr is not ready to fit in with near-realtime, we might as well start pushing down the path. We can work out the names and strategy, but for the moment I have added a 'softCommit' boolean to the commit cmd - if you commit with this set to true, the SolrCore will repen the indexreader using the indexwriter from the updatehandler. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13016874#comment-13016874 ] Yonik Seeley commented on SOLR-2193: NRT finally... sweet! I wonder how this should work with autocommit? Someone may want a soft/reopen autocommit once every x seconds, but still may want a hard flush to stable storage in case I crash commit at some other duration. The other thing that might be cool is a client-specified freshness per request. For example, when they pass in a query, they specify that they need data that's no more than 1 second old... and if it's too old that will trigger a reopen (and block that specific request until the new searcher can be used). The benefit here is that big bulk uploads won't be interrupted if there is no time sensitive query traffic. The downside is that a high latency may be exposed to those requests if they depend on stuff that can take a lot of time the first time (like faceting). Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13016875#comment-13016875 ] Michael McCandless commented on SOLR-2193: -- Fabulous!! Elimination of the IW.close() on Solr's commit, and using Lucene's NRT!! Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: 4.0 Attachments: SOLR-2193.patch The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13015440#comment-13015440 ] Simon Willnauer commented on SOLR-2193: --- bq. I've updated my patch to trunk. hmm maybe I miss something but I don't see a patch on this issue. bq. You can find the most recent work here: https://github.com/markrmiller/Lucene-Solr-Lab/commit/051995d50160e7081854febd471e72b9e81b8a3f this seems to be good stuff why don't you create a branch for it? simon Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: Next The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13015441#comment-13015441 ] Mark Miller commented on SOLR-2193: --- bq. I'll upload a patch after I start lazy creating the IndexWriter. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: Next The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we now do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit (still lazy IW init though). 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. 6. Keep microsharding in mind (maintain logical index as multiple physical indexes) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-2193) Re-architect Update Handler
[ https://issues.apache.org/jira/browse/SOLR-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12924427#action_12924427 ] Mark Miller commented on SOLR-2193: --- I've been playing with a patch the keeps the IndexWriter open always (shares them across core reloads) and drops our internal update locks - so far, all tests pass, but there are still issues to deal with. I'll post the patch once I work a few more things out. Won't cover everything - just a start to explore different ideas. Re-architect Update Handler --- Key: SOLR-2193 URL: https://issues.apache.org/jira/browse/SOLR-2193 Project: Solr Issue Type: Improvement Reporter: Mark Miller Fix For: Next The update handler needs an overhaul. A few goals I think we might want to look at: 1. Cleanup - drop DirectUpdateHandler(2) line - move to something like UpdateHandler, DefaultUpdateHandler 2. Expose the SolrIndexWriter in the api or add the proper abstractions to get done what we know do with special casing: if (directupdatehandler2) success else failish 3. Stop closing the IndexWriter and start using commit. 4. Drop iwAccess, iwCommit locks and sync mostly at the Lucene level. 5. Keep NRT support in mind. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
