[jira] [Commented] (SOLR-6354) Support stats over functions
[ https://issues.apache.org/jira/browse/SOLR-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130901#comment-14130901 ] Hoss Man commented on SOLR-6354: bq. Also, regarding changing the output key: either this was either broken already, or I broke it somehow. Yeah, beyond the initial problem you pointed out about code duplication dealing with where/how the StatsValues instances are constructed, theres also inconsistencies in when/how/if the localparams are parsed. I'm tackling that in SOLR-6507 first. > Support stats over functions > > > Key: SOLR-6354 > URL: https://issues.apache.org/jira/browse/SOLR-6354 > Project: Solr > Issue Type: Sub-task >Reporter: Hoss Man > Attachments: TstStatsComponent.java > > > The majority of the logic in StatsValuesFactory for dealing with stats over > fields just uses the ValueSource API. There's very little reason we can't > generalize this to support computing aggregate stats over any arbitrary > function (or the scores from an arbitrary query). > Example... > {noformat} > stats.field={!func key=mean_rating > mean=true}prod(user_rating,pow(editor_rating,2)) > {noformat} > ...would mean that we can compute a conceptual "rating" for each doc by > multiplying the user_rating field by the square of the editor_rating field, > and then we'd compute the mean of that "rating" across all docs in the set > and return it as "mean_rating" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6354) Support stats over functions
[ https://issues.apache.org/jira/browse/SOLR-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131587#comment-14131587 ] Crawdaddy commented on SOLR-6354: - Excellent - thanks Hoss. Maybe crosstalk, but do you think some of this work would make it easier for us to do stats on scores? Scores mean something in my application and I want to use them in the Stats component. > Support stats over functions > > > Key: SOLR-6354 > URL: https://issues.apache.org/jira/browse/SOLR-6354 > Project: Solr > Issue Type: Sub-task >Reporter: Hoss Man > Attachments: TstStatsComponent.java > > > The majority of the logic in StatsValuesFactory for dealing with stats over > fields just uses the ValueSource API. There's very little reason we can't > generalize this to support computing aggregate stats over any arbitrary > function (or the scores from an arbitrary query). > Example... > {noformat} > stats.field={!func key=mean_rating > mean=true}prod(user_rating,pow(editor_rating,2)) > {noformat} > ...would mean that we can compute a conceptual "rating" for each doc by > multiplying the user_rating field by the square of the editor_rating field, > and then we'd compute the mean of that "rating" across all docs in the set > and return it as "mean_rating" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6354) Support stats over functions
[ https://issues.apache.org/jira/browse/SOLR-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135528#comment-14135528 ] Crawdaddy commented on SOLR-6354: - Nice work Hoss - thank you very much for the patch and the reminder about frange. Will give this a try. > Support stats over functions > > > Key: SOLR-6354 > URL: https://issues.apache.org/jira/browse/SOLR-6354 > Project: Solr > Issue Type: Sub-task >Reporter: Hoss Man > Attachments: SOLR-6354.patch, TstStatsComponent.java > > > The majority of the logic in StatsValuesFactory for dealing with stats over > fields just uses the ValueSource API. There's very little reason we can't > generalize this to support computing aggregate stats over any arbitrary > function (or the scores from an arbitrary query). > Example... > {noformat} > stats.field={!func key=mean_rating > mean=true}prod(user_rating,pow(editor_rating,2)) > {noformat} > ...would mean that we can compute a conceptual "rating" for each doc by > multiplying the user_rating field by the square of the editor_rating field, > and then we'd compute the mean of that "rating" across all docs in the set > and return it as "mean_rating" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6354) Support stats over functions
[ https://issues.apache.org/jira/browse/SOLR-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143512#comment-14143512 ] ASF subversion and git services commented on SOLR-6354: --- Commit 1626856 from hoss...@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1626856 ] SOLR-6354: stats.field can now be used to generate stats over the numeric results of arbitrary functions > Support stats over functions > > > Key: SOLR-6354 > URL: https://issues.apache.org/jira/browse/SOLR-6354 > Project: Solr > Issue Type: Sub-task >Reporter: Hoss Man > Attachments: SOLR-6354.patch, SOLR-6354.patch, SOLR-6354.patch, > TstStatsComponent.java > > > The majority of the logic in StatsValuesFactory for dealing with stats over > fields just uses the ValueSource API. There's very little reason we can't > generalize this to support computing aggregate stats over any arbitrary > function (or the scores from an arbitrary query). > Example... > {noformat} > stats.field={!func key=mean_rating > mean=true}prod(user_rating,pow(editor_rating,2)) > {noformat} > ...would mean that we can compute a conceptual "rating" for each doc by > multiplying the user_rating field by the square of the editor_rating field, > and then we'd compute the mean of that "rating" across all docs in the set > and return it as "mean_rating" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6354) Support stats over functions
[ https://issues.apache.org/jira/browse/SOLR-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14143660#comment-14143660 ] ASF subversion and git services commented on SOLR-6354: --- Commit 1626875 from hoss...@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1626875 ] SOLR-6354: stats.field can now be used to generate stats over the numeric results of arbitrary functions (merge r1626856) > Support stats over functions > > > Key: SOLR-6354 > URL: https://issues.apache.org/jira/browse/SOLR-6354 > Project: Solr > Issue Type: Sub-task >Reporter: Hoss Man > Attachments: SOLR-6354.patch, SOLR-6354.patch, SOLR-6354.patch, > TstStatsComponent.java > > > The majority of the logic in StatsValuesFactory for dealing with stats over > fields just uses the ValueSource API. There's very little reason we can't > generalize this to support computing aggregate stats over any arbitrary > function (or the scores from an arbitrary query). > Example... > {noformat} > stats.field={!func key=mean_rating > mean=true}prod(user_rating,pow(editor_rating,2)) > {noformat} > ...would mean that we can compute a conceptual "rating" for each doc by > multiplying the user_rating field by the square of the editor_rating field, > and then we'd compute the mean of that "rating" across all docs in the set > and return it as "mean_rating" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6354) Support stats over functions
[ https://issues.apache.org/jira/browse/SOLR-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101118#comment-14101118 ] Andy Crossen commented on SOLR-6354: Hey Hoss, I think you mean StatsInfo should do the check you propose? At least, that's where I found I needed to start intercepting this. I have all but the last line in your proposal implemented in StatsInfo.parse in a custom copy of StatsComponent, but having a little trouble seeing how to go from ValueSource -> StatsValues. Can you provide a couple more pointers here? > Support stats over functions > > > Key: SOLR-6354 > URL: https://issues.apache.org/jira/browse/SOLR-6354 > Project: Solr > Issue Type: Sub-task >Reporter: Hoss Man > > The majority of the logic in StatsValuesFactory for dealing with stats over > fields just uses the ValueSource API. There's very little reason we can't > generalize this to support computing aggregate stats over any arbitrary > function (or the scores from an arbitrary query). > Example... > {noformat} > stats.field={!func key=mean_rating > mean=true}prod(user_rating,pow(editor_rating,2)) > {noformat} > ...would mean that we can compute a conceptual "rating" for each doc by > multiplying the user_rating field by the square of the editor_rating field, > and then we'd compute the mean of that "rating" across all docs in the set > and return it as "mean_rating" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6354) Support stats over functions
[ https://issues.apache.org/jira/browse/SOLR-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101414#comment-14101414 ] Hoss Man commented on SOLR-6354: bq. Hey Hoss, I think you mean StatsInfo should do the check you propose? At least, that's where I found I needed to start intercepting this. Hmmm... i guess there's two code paths here that have to be considered? I think i was looking at SimpleStats because that's where the individual {{stats.field}} values are parsed into the "localParams" variable -- so we definitely need to check for a type there, and then do the right thing as far as dealing with the NumericStatsValues in terms of methods like {{SimpleStats.getStatsFields()}} and/or {{SimpleStats.getFieldCacheStats()}} But i think you're right about StatsInfo ... looks like we need to account for it there as well ... i'd need to look over this more closely to understand what's going on there and why bq. ...having a little trouble seeing how to go from ValueSource -> StatsValues. Can you provide a couple more pointers here? If you look at {{StatsValuesFactory.createStatsValues}} and the existing {{AbstractStatsValues}} you'll see that it maintains references to the SchemaField/FieldType of the associated field -- but the meat of the logic is in asking the FieldType for it's ValueSource to then accumulate values from. So what i had in mind was refactoring "field" specific bits of {{AbstractStatsValues}} as needed so that a (new) subclass could be completely field agnostic, and just do the accumulation directly from a VlaueSource passed in (based on the FunctionQParser in most cases) bq. I have all but the last line in your proposal implemented... FYI: no need to "hold back" changes until they are "done" ... yonik's law of patches... bq. A half-baked patch in Jira, with no documentation, no tests and no backwards compatibility is better than no patch at all. > Support stats over functions > > > Key: SOLR-6354 > URL: https://issues.apache.org/jira/browse/SOLR-6354 > Project: Solr > Issue Type: Sub-task >Reporter: Hoss Man > > The majority of the logic in StatsValuesFactory for dealing with stats over > fields just uses the ValueSource API. There's very little reason we can't > generalize this to support computing aggregate stats over any arbitrary > function (or the scores from an arbitrary query). > Example... > {noformat} > stats.field={!func key=mean_rating > mean=true}prod(user_rating,pow(editor_rating,2)) > {noformat} > ...would mean that we can compute a conceptual "rating" for each doc by > multiplying the user_rating field by the square of the editor_rating field, > and then we'd compute the mean of that "rating" across all docs in the set > and return it as "mean_rating" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6354) Support stats over functions
[ https://issues.apache.org/jira/browse/SOLR-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101572#comment-14101572 ] Crawdaddy commented on SOLR-6354: - I did manage to get something working, but it has some problems. I see what you mean on the StatsValuesFactory refactor. Because I was experimenting with this in a copy of StatsComponent (which by the way, is not easy to do!), I ended up not modifying StatsValuesFactory at all. Instead, I wrote a couple inner classes extending NumericStatsValues and FieldType that take a ValueSource as input. In SimpleStats.getStatsFields() and SimpleStats.getFieldCacheStats(), I catch the exception to schema.getField() that is thrown when trying to look up non-existent function fields, and return my custom NSV/FT-based classes - stored in rb._statsInfo - instead. This seems to have broken stat faceting, however, I think since other calls to StatsValuesFactory.createStatsValues outside StatsComponent don't use the same logic. No doubt yours is the better road to travel - I was shooting for quick-n-dirty to see if this was a useful approach to a stats problem. Also, regarding changing the output key: either this was either broken already, or I broke it somehow. Would it be useful for me to upload what I have as a reference point for you or someone else to implement more coherently? I'm not sure I have the bandwidth to pull down a virgin Solr and migrate the changes at this time. > Support stats over functions > > > Key: SOLR-6354 > URL: https://issues.apache.org/jira/browse/SOLR-6354 > Project: Solr > Issue Type: Sub-task >Reporter: Hoss Man > > The majority of the logic in StatsValuesFactory for dealing with stats over > fields just uses the ValueSource API. There's very little reason we can't > generalize this to support computing aggregate stats over any arbitrary > function (or the scores from an arbitrary query). > Example... > {noformat} > stats.field={!func key=mean_rating > mean=true}prod(user_rating,pow(editor_rating,2)) > {noformat} > ...would mean that we can compute a conceptual "rating" for each doc by > multiplying the user_rating field by the square of the editor_rating field, > and then we'd compute the mean of that "rating" across all docs in the set > and return it as "mean_rating" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6354) Support stats over functions
[ https://issues.apache.org/jira/browse/SOLR-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14101604#comment-14101604 ] Hoss Man commented on SOLR-6354: bq. Would it be useful for me to upload what I have as a reference point for you or someone else to implement more coherently? Yes, absolutely ... didn't you see my last comment? bq. A half-baked patch in Jira, with no documentation, no tests and no backwards compatibility is better than no patch at all. > Support stats over functions > > > Key: SOLR-6354 > URL: https://issues.apache.org/jira/browse/SOLR-6354 > Project: Solr > Issue Type: Sub-task >Reporter: Hoss Man > > The majority of the logic in StatsValuesFactory for dealing with stats over > fields just uses the ValueSource API. There's very little reason we can't > generalize this to support computing aggregate stats over any arbitrary > function (or the scores from an arbitrary query). > Example... > {noformat} > stats.field={!func key=mean_rating > mean=true}prod(user_rating,pow(editor_rating,2)) > {noformat} > ...would mean that we can compute a conceptual "rating" for each doc by > multiplying the user_rating field by the square of the editor_rating field, > and then we'd compute the mean of that "rating" across all docs in the set > and return it as "mean_rating" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-6354) Support stats over functions
[ https://issues.apache.org/jira/browse/SOLR-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091545#comment-14091545 ] Hoss Man commented on SOLR-6354: Proposed implementation... * SimpleStats should check the local params of each {{stats.field}} for a "type" param ** type is already treated special in local param parsing as as way to specify the parser type for the body of the string, ie: {{\{!foo\}...}} is just an alias for {{\{!type=foo\}...}} * if "type" param doesn't exist, or is the empty string, treat the param value as a regular field name and get it's value source (just like is done today) * if "type" does exist, do normal QParserPlugin lookup to parse the param value ** if the resulting Query is {{instanceof FunctionQuery}}, cast it and pull out it's ValueSource ** else: wrap the Query in a QueryValueSource * add a new subclass of NumericStatsValues that can be constructed directly with a ValueSource > Support stats over functions > > > Key: SOLR-6354 > URL: https://issues.apache.org/jira/browse/SOLR-6354 > Project: Solr > Issue Type: Sub-task >Reporter: Hoss Man > > The majority of the logic in StatsValuesFactory for dealing with stats over > fields just uses the ValueSource API. There's very little reason we can't > generalize this to support computing aggregate stats over any arbitrary > function (or the scores from an arbitrary query). > Example... > {noformat} > stats.field={!func key=mean_rating > mean=true}prod(user_rating,pow(editor_rating,2)) > {noformat} > ...would mean that we can compute a conceptual "rating" for each doc by > multiplying the user_rating field by the square of the editor_rating field, > and then we'd compute the mean of that "rating" across all docs in the set > and return it as "mean_rating" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org