[jira] [Commented] (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13038507#comment-13038507 ] Jan Høydahl commented on CONNECTORS-19: --- Yea, guess the net effect is about the same if MCF handles the threads or SolrJ does. Guess we could set threadCount=1 and make buffer size configurable. The point of switching to SolrJ would be the assumption that code is more stable and performant. Also SOLR-1565 could make things even faster. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: ManifoldCF Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13038521#comment-13038521 ] Karl Wright commented on CONNECTORS-19: --- That's why this ticket was created - to explore using solrj instead of the homegrown code currently in the connector. However, there are issues we need to consider before solrj would be an option. The guaranteed delivery problem is one such. But also if SolrJ spins up its own threads it might well make it difficult to shut ManifoldCF down properly, depending on how those threads are created. Just as it is better to use an application server's thread pool when you are a web application, the same principles apply for threads created by connectors and their supporting libraries. If you have access to ManifoldCF in Action, you might want to have a look at chapters 5 and 6 for details. However, that does not rule solrj out, it just means we need to be cautious if and when the Solr connector is transitioned to use it. If you want to explore this in detail by all means feel free - patches are definitely welcome. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: ManifoldCF Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13038327#comment-13038327 ] Jan Høydahl commented on CONNECTORS-19: --- Any progress on this? I'd like to see a Solr outputConnector with MultiThread support (StreamingUpdateSolrServer) Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: ManifoldCF Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13038360#comment-13038360 ] Karl Wright commented on CONNECTORS-19: --- The promised patch never materialized. One point, though, is that ManifoldCF is not single-threaded in any case, so you'd be unlikely to gain much in performance by going multithread on an already multi-threaded connector implementation. The current connector can maintain and use as many connections to Solr as you tell it. Memory buffering on the client side also is not a good idea because it violates the basic ManifoldCF principle that you can safely shut down and restart ManifoldCF at any time without loss. Solr also suffers from lack of a guaranteed delivery metaphor, which I've talked to the Solr team about in the past. The Solr commit model currently does not work this way but ManifoldCF really requires it, because without it there is no way to properly implement an incremental crawler. This would mean a significant new Solr feature. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: ManifoldCF Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898859#action_12898859 ] Jettro Coenradie commented on CONNECTORS-19: We have a working solr connector that makes use of solr. This might be a good start. I might need to spend some time to make it run in the lcf build. We have a maven build to package it at the moment. If you are interested, let me know. Than I will spend the time on a patch. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: Lucene Connector Framework Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (CONNECTORS-19) Look into converting SOLR connector to use SolrJ java library
[ https://issues.apache.org/jira/browse/CONNECTORS-19?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12898928#action_12898928 ] Jettro Coenradie commented on CONNECTORS-19: I will have a good look at the dependencies and the functionality. If satisfied, I will supply a patch that other can check as well. Look into converting SOLR connector to use SolrJ java library - Key: CONNECTORS-19 URL: https://issues.apache.org/jira/browse/CONNECTORS-19 Project: Lucene Connector Framework Issue Type: Improvement Components: Lucene/SOLR connector Reporter: Karl Wright Priority: Minor The SOLR connector currently uses its own multipart post code. It might be a good idea to convert it to use the SolrJ client api jar instead. This would require license confirmation, plus research to make sure there are no jar conflicts as a result, with any other connector. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
