Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()
On Jan 5, 2010, at 1:56 AM, Noble Paul നോബിള് नोब्ळ् wrote: This looks like a hack. It currently only uses highlighter for prefetching docs and fields . There is no standard way of other components to take part in this. Possibly, but highlighting is one of the more expensive things to do and making sure the fields are there (and not lazily loaded) is important. Of course, it doesn't help if you want to use Term Vectors w/ highlighter We should either remove this altogether -1. or have a standard way for all components to take part in this. Perhaps a component could register what fields it needs? However, do you have a use case in mind? What component would you like to have leverage this? -Grant
Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()
On Tue, Jan 5, 2010 at 4:52 PM, Grant Ingersoll gsing...@apache.org wrote: On Jan 5, 2010, at 1:56 AM, Noble Paul നോബിള് नोब्ळ् wrote: This looks like a hack. It currently only uses highlighter for prefetching docs and fields . There is no standard way of other components to take part in this. Possibly, but highlighting is one of the more expensive things to do and making sure the fields are there (and not lazily loaded) is important. Of course, it doesn't help if you want to use Term Vectors w/ highlighter We should either remove this altogether -1. or have a standard way for all components to take part in this. Perhaps a component could register what fields it needs? However, do you have a use case in mind? What component would you like to have leverage this? I don't know. But the point is can we have a an interface PrefetchAware (or anything nicer) and components can choose to return the list of fields which it is interested in prefetching. I would like to remove the Strong coupling of QueryComponent on highlighting. -Grant -- - Noble Paul | Systems Architect| AOL | http://aol.com
Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()
2010/1/5 Noble Paul നോബിള് नोब्ळ् noble.p...@corp.aol.com: On Tue, Jan 5, 2010 at 4:52 PM, Grant Ingersoll gsing...@apache.org wrote: On Jan 5, 2010, at 1:56 AM, Noble Paul നോബിള് नोब्ळ् wrote: This looks like a hack. It currently only uses highlighter for prefetching docs and fields . There is no standard way of other components to take part in this. Possibly, but highlighting is one of the more expensive things to do and making sure the fields are there (and not lazily loaded) is important. Of course, it doesn't help if you want to use Term Vectors w/ highlighter We should either remove this altogether -1. or have a standard way for all components to take part in this. Perhaps a component could register what fields it needs? However, do you have a use case in mind? What component would you like to have leverage this? I don't know. But the point is can we have a an interface PrefetchAware (or anything nicer) and components can choose to return the list of fields which it is interested in prefetching. I would like to remove the Strong coupling of QueryComponent on highlighting. Or we can add a method to ResponseBuilder.addPrefetchFields(String[] fieldNames) and SearchComponents can use this in prepare()/process() to express interest in prefetching. -Grant -- - Noble Paul | Systems Architect| AOL | http://aol.com -- - Noble Paul | Systems Architect| AOL | http://aol.com
Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()
On Jan 5, 2010, at 6:52 AM, Noble Paul നോബിള് नोब्ळ् wrote: On Tue, Jan 5, 2010 at 4:52 PM, Grant Ingersoll gsing...@apache.org wrote: On Jan 5, 2010, at 1:56 AM, Noble Paul നോബിള് नोब्ळ् wrote: This looks like a hack. It currently only uses highlighter for prefetching docs and fields . There is no standard way of other components to take part in this. Possibly, but highlighting is one of the more expensive things to do and making sure the fields are there (and not lazily loaded) is important. Of course, it doesn't help if you want to use Term Vectors w/ highlighter We should either remove this altogether -1. or have a standard way for all components to take part in this. Perhaps a component could register what fields it needs? However, do you have a use case in mind? What component would you like to have leverage this? I don't know. But the point is can we have a an interface PrefetchAware (or anything nicer) and components can choose to return the list of fields which it is interested in prefetching. I would like to remove the Strong coupling of QueryComponent on highlighting. Sounds reasonable to me.
Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()
ok I have opened a new issue https://issues.apache.org/jira/browse/SOLR-1702 On Tue, Jan 5, 2010 at 5:50 PM, Grant Ingersoll gsing...@apache.org wrote: On Jan 5, 2010, at 6:52 AM, Noble Paul നോബിള് नोब्ळ् wrote: On Tue, Jan 5, 2010 at 4:52 PM, Grant Ingersoll gsing...@apache.org wrote: On Jan 5, 2010, at 1:56 AM, Noble Paul നോബിള് नोब्ळ् wrote: This looks like a hack. It currently only uses highlighter for prefetching docs and fields . There is no standard way of other components to take part in this. Possibly, but highlighting is one of the more expensive things to do and making sure the fields are there (and not lazily loaded) is important. Of course, it doesn't help if you want to use Term Vectors w/ highlighter We should either remove this altogether -1. or have a standard way for all components to take part in this. Perhaps a component could register what fields it needs? However, do you have a use case in mind? What component would you like to have leverage this? I don't know. But the point is can we have a an interface PrefetchAware (or anything nicer) and components can choose to return the list of fields which it is interested in prefetching. I would like to remove the Strong coupling of QueryComponent on highlighting. Sounds reasonable to me. -- - Noble Paul | Systems Architect| AOL | http://aol.com
Re: Why do we need SolrPluginUtils#optimizePreFetchDocs()
: This looks like a hack. It currently only uses highlighter for : prefetching docs and fields . There is no standard way of other : components to take part in this. It was an optimization that improved the common when it was added, but it predates all of the search component stuff, so that's not much of a suprise that it's not easy for components to use. : Or we can add a method to ResponseBuilder.addPrefetchFields(String[] : fieldNames) and SearchComponents can use this in prepare()/process() : to express interest in prefetching. I would suggest using something like a FieldSelector instead of a String[], so that components wanting all files that match a rule don't have to manifest a large array. I can't put my finger on it, but it feels like there is a larger issue here ... relating to SOLR-1298, and how/when a DocList might/should get manifested as DocumentList ... It would be fairly easy to modify optimizePreFetchDocs to check properties on the ResponseBuilder (via the SolrQueryRequest) to decide which fields to ask for, but ultimately all optimizePreFetchDocs does is ask the SolrIndexSearche to load the doc and then throws it away (relying on the documentCache to have those fields handy for later use). as long as we're changing that, we might as well make it do ... something ... better. -Hoss
Why do we need SolrPluginUtils#optimizePreFetchDocs()
This looks like a hack. It currently only uses highlighter for prefetching docs and fields . There is no standard way of other components to take part in this. We should either remove this altogether or have a standard way for all components to take part in this. -- - Noble Paul | Systems Architect| AOL | http://aol.com
