On 20 June 2014 at 16:47:20, Anže Starič ([email protected]) wrote:
You do not need to implement your own admin commands, bhsearch optimize_index should work. How about the case when the admin switches from one search backend to another, shouldn’t the appropriate index be populated with all existing resources in BH? This is mainly why I was thinking I need to implement my own admin commands. I took a look at your code, and the indexing part seems to be working. I had to change the definition of start_operation [1] so it can be used with the with statement, but the rest worked out of the box. The code is used is pushed to my branch on github [2], if you want to take a look. I used it with the following additions to the trac.ini: [bhsearch] search_backend = SolrBackend solr_server_url = http://localhost:8983/solr/ [components] bhsearch.query_suggestion.* = disabled bhsearch.security.* = disabled Thank you for the link, I took your example and added the config option for providing a Solr server path (I was planning on making this configurable, and not hard-coded, but I wasn’t sure of how to do this). I also made the changes you mentioned. I am currently working on displaying the retrieved Solr results in the interface. The results are currently shown in the interface, but I am working on applying highlighting and faceting. Also, I have a question regarding the meta keyword parsers. How are the DocTypeMetaKeywordParser and the other keyword parsers from bhsearch.query_parser used? I understood in general what the DefaultQueryParser does, however I’m not sure I get how parser plugins are used in Whoosh. I would like to understand the query_parser module better because I used the DefaultQueryParser for parsing the query. I’m not sure if this is a good idea because basically it uses Whoosh for parsing the query, but it was easier for the moment. Should I try to implement my own query parser for Solr? Best, Antonia Anze [1] Changes to start_operation: change def start_operation(self): return None to @contextmanager def start_operation(self): yield [2] https://github.com/astaric/bloodhound/commit/27db226d8e03ffb1c83e5a906b73bfdfe8dc1c25 On Fri, Jun 20, 2014 at 2:45 AM, Antonia Horincar <[email protected]> wrote: > Created Sunburnt query chains for querying Solr; this is the equivalent of > the DefaultQueryParser class that creates queries for Whoosh. Also > implemented the IAdminCommandProvider interface for providing two trac-admin > commands to pre-poluate BH with existing resources, and for optimising the > index. However, the two trac-admin commands doesn’t seem to work for the > moment (when trying to run them, I get a “command not found” error), but I’m > currently trying to find out why this happens. > > On 20 June 2014 at 01:40:08, [email protected] ([email protected]) > wrote: > > Author: ahorincar > Date: Fri Jun 20 00:39:42 2014 > New Revision: 1604058 > > URL: http://svn.apache.org/r1604058 > Log: > Added methods to create Sunburnt query chains > > Added: > bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/admin.py > bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/api.py > Removed: > bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/htdocs/ > bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/templates/ > bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/web_ui.py > Modified: > bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/solr_backend.py > bloodhound/branches/bep_0014_solr/bloodhound_solr/setup.py > > Added: bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/admin.py > URL: > http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/admin.py?rev=1604058&view=auto > > ============================================================================== > > --- bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/admin.py (added) > +++ bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/admin.py Fri Jun > 20 00:39:42 2014 > @@ -0,0 +1,16 @@ > +r"""Administration commands for Bloodhound Solr Search.""" > +from trac.core import Component, implements > +from trac.admin import IAdminCommandProvider > +from bhsolr.api import BloodhoundSolrApi > + > +class BloodhoundSolrSearchAdmin(Component): > + """Bloodhound Solr Search administration component.""" > + implements(IAdminCommandProvider) > + > + # IAdminCommandProvider methods > + def get_admin_commands(self): > + yield ('bhsolr populate_index', '', 'Populate Solr search index', > + None, BloodhoundSolrApi(self.env).populate_index) > + yield ('bhsolr optimize', '', 'Optimize Solr search index', > + None, BloodhoundSearchApi(self.env).optimize) > + > > Added: bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/api.py > URL: > http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/api.py?rev=1604058&view=auto > > ============================================================================== > > --- bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/api.py (added) > +++ bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/api.py Fri Jun > 20 00:39:42 2014 > @@ -0,0 +1,7 @@ > +from bhsolr.search_resources import TicketSearchModel > +from trac.core import Component, implements > + > +class BloodhoundSolrApi(Component): > + def populate_index(self): > + tickets = TicketSearchModel(self.env).get_entries_for_index() > + print tickets > > Modified: > bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/solr_backend.py > URL: > http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/solr_backend.py?rev=1604058&r1=1604057&r2=1604058&view=diff > > ============================================================================== > > --- bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/solr_backend.py > (original) > +++ bloodhound/branches/bep_0014_solr/bloodhound_solr/bhsolr/solr_backend.py > Fri Jun 20 00:39:42 2014 > @@ -7,17 +7,19 @@ from trac.ticket.api import TicketSystem > from bhsearch.search_resources.ticket_search import TicketIndexer > from datetime import datetime > from trac.util.datefmt import utc > +from bhsearch.api import ISearchBackend > +from bhsearch.query_parser import DefaultQueryParser > > UNIQUE_ID = "unique_id" > > -class SolrModel(Component): > +class SolrBackend(Component): > implements(ISearchBackend) > > def __init__(self): > resource_filename = pkg_resources.resource_filename > path = resource_filename(__name__, "schemadoc") > - # self.solr_interface = Solr("http://localhost:8983/solr/";, path + > '/schema.xml').solr_interface > - self.solr_interface = Solr("http://localhost:8983/solr/";).solr_interface > + self.solr_interface = Solr("http://localhost:8983/solr/";).solr_interface # > TODO: Refactor > + self.field_boosts = DefaultQueryParser(self.env).field_boosts > > > def add_doc(self, doc, operation_context=None): > @@ -34,13 +36,44 @@ class SolrModel(Component): > self.solr_interface.delete(unique_id) > > > - def optimize(): > + def optimize(self): > self.solr_interface.optimize() > > + def query(self, query, query_string, sort = None, fields = None, filter = > None, > + facets = None, pagenum = 1, pagelen = 20, highlight = False, > + highlight_fields = None, context = None): > + > + tokens = set([token.text for token in query.all_tokens()]) > + final_query_chain = None > + for token in tokens: > + token_query_chain = self._search_fields_for_token(token) > + if final_query_chain == None: > + final_query_chain = token_query_chain > + else: > + final_query_chain |= token_query_chain > + > + print final_query_chain > + return None > > - def query(self, query): > - self.solr_instance.solr_interface.query(query).execute() > + def is_index_outdated(self): > + return False > > + def recreate_index(self): > + return True > + > + def start_operation(self): > + return None > + > + def _search_fields_for_token(self, token): > + query_chain = None > + for field, boost in self.field_boosts.iteritems(): > + field_token_dict = {field: token} > + if query_chain == None: > + query_chain = self.solr_interface.Q(**field_token_dict)**boost > + else: > + query_chain |= self.solr_interface.Q(**field_token_dict)**boost > + > + return query_chain > > def _reformat_doc(self, doc): > for key, value in doc.items(): > @@ -75,12 +108,7 @@ class SolrModel(Component): > else: > return u"%s:%s" % (doc_type, doc_id) > > -if __name__ == '__main__': > - env = > trac.env.Environment("/Users/antonia/Documents/Code/bloodhound/installer/bloodhound/environments/main") > > - db_connection = env.get_db_cnx() > - cursor = db_connection.cursor() > - a = cursor.execute("select * from ticket") > - ticket = a.fetchall()[0] > + def getInstance(self): > + return self.solr_interface > + > > - # for result in si.query(name="Ticket").execute(): > - # print result > > Modified: bloodhound/branches/bep_0014_solr/bloodhound_solr/setup.py > URL: > http://svn.apache.org/viewvc/bloodhound/branches/bep_0014_solr/bloodhound_solr/setup.py?rev=1604058&r1=1604057&r2=1604058&view=diff > > ============================================================================== > > --- bloodhound/branches/bep_0014_solr/bloodhound_solr/setup.py (original) > +++ bloodhound/branches/bep_0014_solr/bloodhound_solr/setup.py Fri Jun 20 > 00:39:42 2014 > @@ -1,6 +1,6 @@ > from setuptools import setup, find_packages > > -PKG_INFO = {'bhsolr': ['htdocs/*.*', 'templates/*', 'schemadoc/*.xml'], > +PKG_INFO = {'bhsolr': ['schemadoc/*.xml'], > 'bhsolr.search_resources' : [], > } > > @@ -8,9 +8,9 @@ PKG_INFO = {'bhsolr': ['htdocs/*.*', 'te > > ENTRY_POINTS = { > 'trac.plugins': [ > - 'bhsolr.web_ui = bhsolr.web_ui', > 'bhsolr.api = bhsolr.api', > 'bhsolr.solr = bhsolr.solr', > + 'bhsolr.admin = bhsolr.admin', > 'bhsolr.solr_backend = bhsolr.solr_backend', > 'bhsolr.search_resources.ticket_search = > bhsolr.search_resources.ticket_search', > 'bhsolr.search_resources.milestone_search = > bhsolr.search_resources.milestone_search', > >
