[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223197#comment-13223197 ] Ryo Hideno commented on SOLR-1566: -- Please do not forget QuerySenderListener. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897544#action_12897544 ] Grant Ingersoll commented on SOLR-1566: --- I know this sounds simple and maybe I'm missing something, but why can't we have a simple data structure that is attached to the SolrResponse and then in the ResponseWriters, (pretty much where it adds the Score onto the item) it looks up the document in the data structure (it's likely a Map where the key is the doc id and the value is a list of values to add to the doc) and then adds the fields, optionally marking them as "pseudo" or annotative fields? I can't imagine the memory usage would get out of hand unless you are retrieving a large number of docs (but then you have memory issues anyway) and it is short-lived so it likely never proceeds out of Eden memory anyway. Everybody already has access to the SolrResponse to some extent. For those who want, we could likely make it a configuration item such that even the holding Map isn't created, but that likely isn't too big of a deal. Thoughts? > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897645#action_12897645 ] Chris Male commented on SOLR-1566: -- Thats basically where I was headed with SOLR-1298. Although I still believe that having them added as fields to the Documents properly, whether it be at the SolrIndexSearcher level or somewhere higher, means we can more easily use the values in other areas without having to write custom code in each place. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897701#action_12897701 ] Grant Ingersoll commented on SOLR-1566: --- bq. Thats basically where I was headed with SOLR-1298 Hmm, missed that, let me check it out again. bq. means we can more easily use the values in other areas True, but Solr doesn't really materialize the Document until the end anyway, so all you are doing is creating the store anyway. I've got a basic implementation already that simply adds a small storage piece onto SolrQueryResponse. It is a Map> where a DocAnnotation is a String name and a Object value. At RespWriting time, it simply adds the objects as fields onto the appropriate documents. The key to the Map is the Lucene doc id. Not sure if that is ideal, but I figure anyone adding to a document would have that handle, but that remains to be seen. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897704#action_12897704 ] Chris Male commented on SOLR-1566: -- bq. I've got a basic implementation already that simply adds a small storage piece onto SolrQueryResponse. It is a Map> where a DocAnnotation is a String name and a Object value. At RespWriting time, it simply adds the objects as fields onto the appropriate documents. The key to the Map is the Lucene doc id. Not sure if that is ideal, but I figure anyone adding to a document would have that handle, but that remains to be seen. Sounds reasonable to me. Except I still fear that by creating the DocAnnotation idea we are now adding another construct to Documents. Sure, as you say, Solr doesn't materialize Documents till it needs to write them out, but still a Document consists only of Fields. Now they consist of Fields and DocAnnotations. Its not a big issue I suppose, just wondering whether its needed. Perhaps a simple renaming to MetadataField or AnnotatedField or something would suffice. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897705#action_12897705 ] Grant Ingersoll commented on SOLR-1566: --- bq. Now they consist of Fields and DocAnnotations. Nah, they don't really. It's just sdoc.addField(annot.name, annot.value) The only thing that is missing in that is it might be nice to be able to mark that field as being a "pseudo" field. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897709#action_12897709 ] Chris Male commented on SOLR-1566: -- bq. Nah, they don't really. It's just sdoc.addField(annot.name, annot.value) Ah okay, super! bq. The only thing that is missing in that is it might be nice to be able to mark that field as being a "pseudo" field. That would be nice. Perhaps we can expand SolrDocument to include a list of pseudo fields? If that needs to be converted into a lucene Document then it can be merged with the rest. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897711#action_12897711 ] Grant Ingersoll commented on SOLR-1566: --- I'd add, however, that it is no different from adding the score. Also, I don't know that marking the field is required, as, IMO, the Document that is written out is intended to be "truth" as far as Solr is concerned, not as far as the Lucene index is concerned. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897713#action_12897713 ] Chris Male commented on SOLR-1566: -- Would you change how score is handled so its like other pseudo fields? > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897714#action_12897714 ] Grant Ingersoll commented on SOLR-1566: --- Then again, there are things like sorting, etc. that we may or may not want to support, so you wouldn't want the app dev. to expect to sort by a pseudo field (or maybe you do) > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897716#action_12897716 ] Chris Male commented on SOLR-1566: -- If we think in terms of spatial distances, then it might be useful to be able to sort by the pseudo field? I know we currently resolve that issue by sorting by function query, but if you already calculate the distance in some other spatial process, might as well be able to sort on it? > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897717#action_12897717 ] Grant Ingersoll commented on SOLR-1566: --- bq. Would you change how score is handled so its like other pseudo fields? I doubt it. It is a bit different, I think, than other pseudo-fields. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897721#action_12897721 ] Grant Ingersoll commented on SOLR-1566: --- bq. If we think in terms of spatial distances, then it might be useful to be able to sort by the pseudo field? I know we currently resolve that issue by sorting by function query, but if you already calculate the distance in some other spatial process, might as well be able to sort on it? Yeah, that is true, but sorting is such a low level operation and this issue is about any component doing it, no matter where in the list it occurs. I just don't see how to make the two sync up w/o having to resort and that would be a perf. killer in many ways (CPU and memory). IMO, I think this piece of functionality should be about "marking up" the result set that is going to be returned. Of course, even with the approach I propose, it has the potential to be a memory killer if done wrong. For instance, I doubt you'd want to stuff in the Map every single distance you calculate, you really only want to add those items that you know are going to be in the final result list. Perhaps the alternate way is to provide a call back mechanism at Document binding time (i.e. the ResponseWriter) where it gives some Interface the document and then the implementation can add what it wants per the application. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897729#action_12897729 ] Chris Male commented on SOLR-1566: -- bq. Yeah, that is true, but sorting is such a low level operation and this issue is about any component doing it, no matter where in the list it occurs. I just don't see how to make the two sync up w/o having to resort and that would be a perf. killer in many ways (CPU and memory). IMO, I think this piece of functionality should be about "marking up" the result set that is going to be returned. I concur. I think its best not to do too much stuff in this issue. If we get pseudo fields being written out, then later on we can address advanced usages. bq. Of course, even with the approach I propose, it has the potential to be a memory killer if done wrong. For instance, I doubt you'd want to stuff in the Map every single distance you calculate, you really only want to add those items that you know are going to be in the final result list. Yup, this is the crux of the issue. bq. Perhaps the alternate way is to provide a call back mechanism at Document binding time (i.e. the ResponseWriter) where it gives some Interface the document and then the implementation can add what it wants per the application. I just wrote something about how I didn't see how that addressed the problem, but then it clicked what you are getting at. Through this Callback you can then re-calculate the distances for your page of results and recalculating 10 distances is a nominal cost. I think thats a really great idea, but I wonder if that shouldn't be an additional process to what you've already made? For spatial it makes perfect sense, but I wonder whether its appropriate for other use cases? I'm trying to imagine some use case where the value can only be captured as part of some convoluted query or analysis process. Consequently you want to store it when you can and not at binding time. But maybe a) those use cases dont exist and b) We should leave it up to the user to create a storage system that hooks into the callback. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897749#action_12897749 ] Grant Ingersoll commented on SOLR-1566: --- bq. but I wonder if that shouldn't be an additional process to what you've already made? Totally agree. bq. b) We should leave it up to the user to create a storage system that hooks into the callback. Yep, that would work too. Heck, it would even allow one to materialize stuff from a DB or Cassandra, etc. We'd probably want to be able to do bulk operations if that is the case. But again, that's getting ahead of ourselves, I think. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897750#action_12897750 ] Grant Ingersoll commented on SOLR-1566: --- The biggest problem here, is the ResponseWriters are a tangled web and it is not always clear they have access to the SolrQueryResponse at the right place, but maybe I'm not understanding them fully yet. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897824#action_12897824 ] Chris Male commented on SOLR-1566: -- Sounds like a good time to untangle the web. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12897879#action_12897879 ] Grant Ingersoll commented on SOLR-1566: --- Hmm, so digging in a bit more here and looking at Noble's patch, I think what I have, Noble has and what Chris has are very similar. The big diff is Chris and Noble hook into the SolrIndexSearcher, whereas what I was intending to do was to hook into the response writers. As mentioned in the prev. comment, this is a bit tedious, such that it likely does make sense to hook into the SolrIndexSearcher. Then again, the SolrQueryRequest already has a context object on it (which is what Chris' patch uses, I think), so maybe we should just use that along w/ the Resp. Writers. Argh. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011221#comment-13011221 ] Ryan McKinley commented on SOLR-1566: - Yonik SOLR-1566_parsing.patch looks good -- should be able to replace the 'ReturnFields' class in my patch when it is ready to go > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-gsi.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[
https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011264#comment-13011264
]
Yonik Seeley commented on SOLR-1566:
I've tried to go over my brainstorming notes and pull out some stuff that might
make sense (as opposed to the stuff that no one will understand w/o a lot more
context, like why approach X or approach Y won't work).
* Know when not to load any stored fields at all (fields could all come from
pseudofields, including external fields)
** example: fl={!func key=id}id// use the fieldcache to load an id
(important option for distributed search)
** example: fl={!func key=id},mul(popularity,editorial_score)&rows=10
// stream the entire thing
* General transformer class: something that adds, changes, or removes fields
based on an external/database lookup
** may need to work in batches of external ids (a getNextDocument() iterator
pattern could allow
buffering and then an external request with a bunch of ids)
** probably wants to work in batches of SolrDocument, not docids
** may need to inspect the FieldsSpec to see what fields were requested
*** ask: did the request ask for field "x" that I am responsible for?
*** ask: did the request *specifically* ask for field "x"
* function query that needs to work in-order for good efficiency (query/scorer
is one example)
** need to work in potentially big blocks (100 docids at a time), sort the
ids then retrieve values in order
** don't want to instantiate all SolrDocuments at once... just keep a
float[100] or Object[100]
** if multiple function queries, don't make each sort it's own list of
docids, share somehow?
*** the sharing mechanism could just be a private implementation to this
type of transformer
* transformers that need the Index... don't make each one look up the right
AtomicReaderContext, share it somehow
* external fields should work with highlighting (eventually)
** needs to be possible for highlighting to work on SolrDocuments and only
run the transformers once... replacing the original DocList with something else
* different DocLists need different transformer lists
** example: multiple DocLists as the result of grouping
** example: allow overriding "fl" per grouping command
* need more than one level of pseudo-fields...
** example: user may specify fl=add(a,b)... that's "global" unless
overridden
but the grouping code may want to add the _group_ pseudo-field when using
the flattened/simple format
> Allow components to add fields to outgoing documents
>
>
> Key: SOLR-1566
> URL: https://issues.apache.org/jira/browse/SOLR-1566
> Project: Solr
> Issue Type: New Feature
> Components: search
>Reporter: Noble Paul
>Assignee: Grant Ingersoll
> Fix For: Next
>
> Attachments: SOLR-1566-DocTransformer.patch,
> SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch,
> SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch,
> SOLR-1566-DocTransformer.patch, SOLR-1566-gsi.patch, SOLR-1566-rm.patch,
> SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch,
> SOLR-1566-rm.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch,
> SOLR-1566.patch, SOLR-1566_parsing.patch
>
>
> Currently it is not possible for components to add fields to outgoing
> documents which are not in the the stored fields of the document. This makes
> it cumbersome to add computed fields/metadata .
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011359#comment-13011359 ] Ryan McKinley commented on SOLR-1566: - Yonik, i took your patch and appled to trunk in: SOLR-2444 I think it makes sense to start a new issue for how the params are parsed since it looks like a lot is going on there > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-gsi.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[
https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018654#comment-13018654
]
Yonik Seeley commented on SOLR-1566:
Hmmm, I think I just hit another issue:
http://localhost:8983/solr/browse
{code}
java.lang.ClassCastException: org.apache.solr.response.ResultContext cannot be
cast to org.apache.solr.common.SolrDocumentList
at org.apache.solr.response.PageTool.(PageTool.java:46)
at
org.apache.solr.response.VelocityResponseWriter.write(VelocityResponseWriter.java:62)
{code}
> Allow components to add fields to outgoing documents
>
>
> Key: SOLR-1566
> URL: https://issues.apache.org/jira/browse/SOLR-1566
> Project: Solr
> Issue Type: New Feature
> Components: search
>Reporter: Noble Paul
> Fix For: 4.0
>
> Attachments: SOLR-1566-DocTransformer.patch,
> SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch,
> SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch,
> SOLR-1566-DocTransformer.patch, SOLR-1566-gsi.patch, SOLR-1566-rm.patch,
> SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch,
> SOLR-1566-rm.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch,
> SOLR-1566.patch, SOLR-1566_parsing.patch
>
>
> Currently it is not possible for components to add fields to outgoing
> documents which are not in the the stored fields of the document. This makes
> it cumbersome to add computed fields/metadata .
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023419#comment-13023419 ] Yonik Seeley commented on SOLR-1566: bq. Somewhere we lost some formatting goodness. I just committed a small fix for this. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152891#comment-13152891 ] Grant Ingersoll commented on SOLR-1566: --- Ryan, I see http://wiki.apache.org/solr/DocTransformers has some pretty thin details on this stuff. How does one actually invoke it at Query Time? I see how to configure it. Is it something like fl=id,score,[myTransformer ... ] ? That's what it seems like based on the ReturnFields parsing code. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152892#comment-13152892 ] Grant Ingersoll commented on SOLR-1566: --- Debugging a bit shows that this is indeed how it is done. I will add it to the docs. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152928#comment-13152928 ] Ryan McKinley commented on SOLR-1566: - I fixed it up a little bit (at least using the right syntax) and added a link to: http://wiki.apache.org/solr/CommonQueryParameters#Transformers > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178856#comment-13178856 ] Antony Stubbs commented on SOLR-1566: - Are there any tests writing for any of the doctransformer architecture? I can't find any. I'm looking for something to base tests for a new regex transformer and highlighting augmenting transformer.. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178874#comment-13178874 ] Hoss Man commented on SOLR-1566: Antony: I don't see any specific "unit" tests of the internals, but if you look at all the svn commits associated with this issue there is at least one example of doing functional tests to ensure that the transformers are called at the output is included in the response documents... https://svn.apache.org/viewvc/lucene/dev/trunk/solr/src/test/org/apache/solr/client/solrj/SolrExampleTests.java?r1=1085450&r2=1085449&view=diff&pathrev=1085450 > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178878#comment-13178878 ] Grant Ingersoll commented on SOLR-1566: --- The QueryElevationComponentTest has some examples as well. I thought there were tests in other places. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178908#comment-13178908 ] Ryan McKinley commented on SOLR-1566: - See Also: * http://wiki.apache.org/solr/DocTransformers * http://wiki.apache.org/solr/CommonQueryParameters#fl > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178922#comment-13178922 ] Ryan McKinley commented on SOLR-1566: - There are some tests in: [SolrExampleTests.java|https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleTests.java]#testAugmentFields() and [TestPseudoReturnFields.java|https://svn.apache.org/repos/asf/lucene/dev/trunk/solr/core/src/test/org/apache/solr/search/TestPseudoReturnFields.java] > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[
https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178926#comment-13178926
]
Antony Stubbs commented on SOLR-1566:
-
Thanks, that's a start. I'm specifically looking to access the highlighting
results, but I can't see how I can. The highlighting results are added to the
ResponseBuilder.Response directly in HighlightingComponent:
rb.rsp.add("highlighting", sumData);. I want to be able to extract that
highlighting component and do some processing on it in my DocTransformer..
> Allow components to add fields to outgoing documents
>
>
> Key: SOLR-1566
> URL: https://issues.apache.org/jira/browse/SOLR-1566
> Project: Solr
> Issue Type: New Feature
> Components: search
>Reporter: Noble Paul
>Assignee: Ryan McKinley
> Fix For: 4.0
>
> Attachments: SOLR-1566-DocTransformer.patch,
> SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch,
> SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch,
> SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch,
> SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch,
> SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch,
> SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch
>
>
> Currently it is not possible for components to add fields to outgoing
> documents which are not in the the stored fields of the document. This makes
> it cumbersome to add computed fields/metadata .
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178938#comment-13178938 ] Ryan McKinley commented on SOLR-1566: - To do highlighting in a Transformer, you may not want/need to use the Highlighting component at all. You could do it directly in the Transformer -- this way you have direct access to the output and can avoid building the Map that gets added to the response. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13179092#comment-13179092 ] Antony Stubbs commented on SOLR-1566: - Thanks Ryan.. I was trying to get access to the highlighting results, rather than run it from the transformer - doesn't seem right. But it could be a start.. I think I saw an access point for that... > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Ryan McKinley > Fix For: 4.0 > > Attachments: SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-DocTransformer.patch, > SOLR-1566-DocTransformer.patch, SOLR-1566-PageTool.patch, > SOLR-1566-gsi.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566_parsing.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902456#action_12902456 ] Grant Ingersoll commented on SOLR-1566: --- I think we all have generally worked around the same issues here, between this and SOLR-1298. I guess we just need to pick some names and work it out. One thing about this last patch (and mine, I think) is that perhaps we should just put the augmenter on the Request. That way, you don't have to add the response in a bunch of places. Besides, in my mind anyway, you are requesting augmentation via the Augmenter provided. Also, I'm not sure why StdAugmenter is instantiated in SolrCore. Wouldn't we want to allow for that to be driven by some user implementations? Perhaps, since there are a few of us w/ eyes on this, we should first try to tackle the ResponseWriter mess. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566-gsi.patch, SOLR-1566-rm.patch, > SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915529#action_12915529 ] Bill Bell commented on SOLR-1566: - OK. Do we have something that works? Near works? Are we going with this or SOLR-1298 ? We need a decision ? If not defined in schema.xml we could just add a parameter: fi=distance:hsin(6500, score, 39, -109) FI = Field Injection (1 or more). It could be handled in the responseWriter. Later adding it to schema.xml can be tackled? Thanks. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566-gsi.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[
https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004227#comment-13004227
]
Ryan McKinley commented on SOLR-1566:
-
I just added a path that *almost* works. Rather then bang my head on it some
more, i think some feedback would be great.
Originally, I hoped to clean up the ResponeWriter mess, and make a single place
that would 'augment' docs -- but there are *so* many micro optimizations for
each format that approach seems difficult.
Instead I went for an approach the moves Set returnFields to a class
that knows more then just a list of strings. This holds an augmenter that can
add to SolrDocument or write to the response.
Now you can make a request like:
http://localhost:8983/solr/select?q=*:*&fl=id,score,_docid_,_shard
and get back a response
{code}
"response":{"numFound":17,"start":0,"maxScore":1.0,"docs":[
{
"id":"GB18030TEST",
"score":1.0,
"_docid_":0,
"_shard_":"getshardid???"},
{
"id":"SP2514N",
"score":1.0,
"_docid_":1,
"_shard_":"getshardid???"},
{
"id":"6H500F0",
"features":[
"SATA 3.0Gb/s, NCQ",
"8.5ms seek",
"16MB cache"],
"score":1.0,
"_docid_":2,
"_shard_":"getshardid???"},
...
{code}
right now, _docid_ just returns the lucene docid, and _shard_ returns a
constant string saying "getshardid???"
The distributed tests (BasicDistributedZkTest, and TestDistributedSearch) don't
pass, but everything else does. I will wait for general feedback before trying
to track that down.
Also looking at SOLR-1298, I would love some feedback on how we could read
function queries and ideally used ones that are already defined elsewhere.
feedback welcome!
> Allow components to add fields to outgoing documents
>
>
> Key: SOLR-1566
> URL: https://issues.apache.org/jira/browse/SOLR-1566
> Project: Solr
> Issue Type: New Feature
> Components: search
>Reporter: Noble Paul
>Assignee: Grant Ingersoll
> Fix For: Next
>
> Attachments: SOLR-1566-gsi.patch, SOLR-1566-rm.patch,
> SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch,
> SOLR-1566-rm.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch,
> SOLR-1566.patch
>
>
> Currently it is not possible for components to add fields to outgoing
> documents which are not in the the stored fields of the document. This makes
> it cumbersome to add computed fields/metadata .
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004240#comment-13004240 ] Yonik Seeley commented on SOLR-1566: Hey Ryan, I've also been working on this recently, trying to keep all of the various use-cases in mind (it's difficult!). It's all just been about brainstorming on the back-end (the chain of things that can modify documents), but I also have code that parses multiple "fl" params, including field globbing and function queries. It doesn't *implement* those yet, just allows for their specification. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566-gsi.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[ https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004354#comment-13004354 ] Bill Bell commented on SOLR-1566: - Here is one use case: - Return geodist() as a field in the results. > Allow components to add fields to outgoing documents > > > Key: SOLR-1566 > URL: https://issues.apache.org/jira/browse/SOLR-1566 > Project: Solr > Issue Type: New Feature > Components: search >Reporter: Noble Paul >Assignee: Grant Ingersoll > Fix For: Next > > Attachments: SOLR-1566-gsi.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch, > SOLR-1566-rm.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch, > SOLR-1566.patch > > > Currently it is not possible for components to add fields to outgoing > documents which are not in the the stored fields of the document. This makes > it cumbersome to add computed fields/metadata . -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[
https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004636#comment-13004636
]
Ryan McKinley commented on SOLR-1566:
-
Here is a different approach that allows the whole Document to be transformed
rather then just letting you add fields to the response. The basic interface
is now:
{code:java}
public abstract class DocTransformer
{
public abstract void transform(SolrDocument doc, int docid, Float score);
}
{code}
Some notes about the patch -- many tests fail because this does not support the
old xml style where multivalued fields show up as a single item.
> Allow components to add fields to outgoing documents
>
>
> Key: SOLR-1566
> URL: https://issues.apache.org/jira/browse/SOLR-1566
> Project: Solr
> Issue Type: New Feature
> Components: search
>Reporter: Noble Paul
>Assignee: Grant Ingersoll
> Fix For: Next
>
> Attachments: SOLR-1566-DocTransformer.patch, SOLR-1566-gsi.patch,
> SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch,
> SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566.patch, SOLR-1566.patch,
> SOLR-1566.patch, SOLR-1566.patch
>
>
> Currently it is not possible for components to add fields to outgoing
> documents which are not in the the stored fields of the document. This makes
> it cumbersome to add computed fields/metadata .
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] Commented: (SOLR-1566) Allow components to add fields to outgoing documents
[
https://issues.apache.org/jira/browse/SOLR-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004800#comment-13004800
]
Ryan McKinley commented on SOLR-1566:
-
Updated patch. This changes the the API to:
{code:java}
public abstract class DocTransformer
{
public void setContext( TransformContext context ) {}
public abstract void transform(SolrDocument doc, int docid);
}
{code}
This will let the TransformContext hold objects that may be useful for filling
up the SolrDocument later.
For example, the DocIterator is included in TransformContext and that is used
to fill in the score (rather then passing Float score to every transformer).
Another example would be if we have the Query object and want to add 'explain'
info directly to the document.
Bill -- re geodist(), yes that is my real motivation for this! see SOLR-1298
Something is still weird with the Distributed search stuff, but figure I should
get feedback on general approach before worrying about that.
> Allow components to add fields to outgoing documents
>
>
> Key: SOLR-1566
> URL: https://issues.apache.org/jira/browse/SOLR-1566
> Project: Solr
> Issue Type: New Feature
> Components: search
>Reporter: Noble Paul
>Assignee: Grant Ingersoll
> Fix For: Next
>
> Attachments: SOLR-1566-DocTransformer.patch,
> SOLR-1566-DocTransformer.patch, SOLR-1566-gsi.patch, SOLR-1566-rm.patch,
> SOLR-1566-rm.patch, SOLR-1566-rm.patch, SOLR-1566-rm.patch,
> SOLR-1566-rm.patch, SOLR-1566.patch, SOLR-1566.patch, SOLR-1566.patch,
> SOLR-1566.patch
>
>
> Currently it is not possible for components to add fields to outgoing
> documents which are not in the the stored fields of the document. This makes
> it cumbersome to add computed fields/metadata .
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
