[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[ https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012681#comment-18012681 ] David Smiley commented on SOLR-5707: That was a heroic fix [~ichattopadhyaya] after the Lucene 10 upgrade; thanks! That feature is very fresh and still in my mind... feel free to just DM me for assistance porting code I recently wrote. It's a shame the fix involves so much extra code; I'm sure if Solr moved off ValueSource completely, it would have been a non-issue. I thought I created an issue for that but I suppose not. Any way, Such a thing would be Solr 11. We can also seek improvements to ValueSource to help reduce our work-around here. One thing I've thought might help is a way to unwrap a ValueSource to an underlying DoubleValuesSource. > Lucene Expressions in Solr > -- > > Key: SOLR-5707 > URL: https://issues.apache.org/jira/browse/SOLR-5707 > Project: Solr > Issue Type: New Feature >Reporter: Ryan Ernst >Assignee: David Smiley >Priority: Major > Labels: pull-request-available > Fix For: 9.9 > > Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Expressions should be available for use in Solr. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[ https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18012624#comment-18012624 ] ASF subversion and git services commented on SOLR-5707: --- Commit a31b9973f19e3b50352f1263c31bcf0071a03a3e in solr's branch refs/heads/main from Ishan Chattopadhyaya [ https://gitbox.apache.org/repos/asf?p=solr.git;h=a31b9973f19 ] SOLR-17631, SOLR-5707: Fixing ExpressionValueSourceParser's sort with scores problem in Lucene 10 > Lucene Expressions in Solr > -- > > Key: SOLR-5707 > URL: https://issues.apache.org/jira/browse/SOLR-5707 > Project: Solr > Issue Type: New Feature >Reporter: Ryan Ernst >Assignee: David Smiley >Priority: Major > Labels: pull-request-available > Fix For: 9.9 > > Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Expressions should be available for use in Solr. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[ https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18005033#comment-18005033 ] ASF subversion and git services commented on SOLR-5707: --- Commit 615258a6fe5fc686cde1bf3b2c23b7096b81d92e in solr's branch refs/heads/branch_9_9 from Kevin Risden [ https://gitbox.apache.org/repos/asf?p=solr.git;h=615258a6fe5 ] SOLR-5707: Lucene Expressions via ExpressionValueSourceParser (#1244) New ExpressionValueSourceParser that allows custom function queries / VSPs to be defined in a subset of JavaScript, pre-compiled, and that which can access the score and fields. It's powered by the Lucene Expressions module. ValueSourceAugmenter: score propagation - Co-authored-by: Chris Hostetter Co-authored-by: David Smiley Co-authored-by: Ryan Ernst (cherry picked from commit c3b5f57cd068a362a5992e57f855e90c34cf329a) > Lucene Expressions in Solr > -- > > Key: SOLR-5707 > URL: https://issues.apache.org/jira/browse/SOLR-5707 > Project: Solr > Issue Type: New Feature >Reporter: Ryan Ernst >Priority: Major > Labels: pull-request-available > Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch > > Time Spent: 3h > Remaining Estimate: 0h > > Expressions should be available for use in Solr. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[ https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18005027#comment-18005027 ] ASF subversion and git services commented on SOLR-5707: --- Commit 3b9c7819ddeb7cde1ac96acab2df9a5c02d4b4e1 in solr's branch refs/heads/branch_9x from Kevin Risden [ https://gitbox.apache.org/repos/asf?p=solr.git;h=3b9c7819dde ] SOLR-5707: Lucene Expressions via ExpressionValueSourceParser (#1244) New ExpressionValueSourceParser that allows custom function queries / VSPs to be defined in a subset of JavaScript, pre-compiled, and that which can access the score and fields. It's powered by the Lucene Expressions module. ValueSourceAugmenter: score propagation - Co-authored-by: Chris Hostetter Co-authored-by: David Smiley Co-authored-by: Ryan Ernst (cherry picked from commit c3b5f57cd068a362a5992e57f855e90c34cf329a) > Lucene Expressions in Solr > -- > > Key: SOLR-5707 > URL: https://issues.apache.org/jira/browse/SOLR-5707 > Project: Solr > Issue Type: New Feature >Reporter: Ryan Ernst >Priority: Major > Labels: pull-request-available > Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch > > Time Spent: 3h > Remaining Estimate: 0h > > Expressions should be available for use in Solr. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[ https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18005012#comment-18005012 ] ASF subversion and git services commented on SOLR-5707: --- Commit c3b5f57cd068a362a5992e57f855e90c34cf329a in solr's branch refs/heads/main from Kevin Risden [ https://gitbox.apache.org/repos/asf?p=solr.git;h=c3b5f57cd06 ] SOLR-5707: Lucene Expressions via ExpressionValueSourceParser (#1244) New ExpressionValueSourceParser that allows custom function queries / VSPs to be defined in a subset of JavaScript, pre-compiled, and that which can access the score and fields. It's powered by the Lucene Expressions module. ValueSourceAugmenter: score propagation - Co-authored-by: Chris Hostetter Co-authored-by: David Smiley Co-authored-by: Ryan Ernst > Lucene Expressions in Solr > -- > > Key: SOLR-5707 > URL: https://issues.apache.org/jira/browse/SOLR-5707 > Project: Solr > Issue Type: New Feature >Reporter: Ryan Ernst >Priority: Major > Labels: pull-request-available > Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch > > Time Spent: 3h > Remaining Estimate: 0h > > Expressions should be available for use in Solr. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[ https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17985989#comment-17985989 ] David Smiley commented on SOLR-5707: I've substantially updated the PR to go this route. It's much simpler; no recursion check needed. It can access positional arguments via $1, $2, etc., and those arguments can be whatever (constants, fields, other function queries). The expression itself can't access other VSPs but it's not a big deal with the positional argument support. The PR is nearly ready to go but I'm waiting for a bug fix in 9.12.2 that substantially improves the functionality as it allows score access in a sort, which is super useful for boosting in exotic ways that are not performant or impossible otherwise. > Lucene Expressions in Solr > -- > > Key: SOLR-5707 > URL: https://issues.apache.org/jira/browse/SOLR-5707 > Project: Solr > Issue Type: New Feature >Reporter: Ryan Ernst >Priority: Major > Labels: pull-request-available > Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Expressions should be available for use in Solr. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[
https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17985630#comment-17985630
]
David Smiley commented on SOLR-5707:
I updated the PR with a new Solr Ref Guide page. As I look at it and think
about this, I don't think it's ideal for there to be one registration of this
value source parser that lists named expressions within it. Instead, I think
each named expression should itself be a ValueSource, and ideally it would then
have access to all other value sources (i.e. those defined by Solr and custom
ones). So instead of
{code:xml}
sqrt(popularity)
log(price)
sqrt_popularity / log_price
{code}
We'd have 3 separate ones. Here's the first:
{code:xml}
sqrt(popularity)
{code}
I think this is much easier understand -- basically it allows someone to define
a custom ValueSource merely by putting a small JavaScript expression. I think
this would deliver the best user experience, and integration with other
ValueSources shouldn't be an afterthought -- it's expected/natural. There may
be some challenges to late-bind/compile since I don't think it can be done
early, since at that time, not all VSPs are even known. At first we could just
convert any function call to a lookup of a ValueSource that we convert to a
DoubleValuesSource. Ideally we could detect if it could be unwrapped but that
could come later with a small Lucene addition.
> Lucene Expressions in Solr
> --
>
> Key: SOLR-5707
> URL: https://issues.apache.org/jira/browse/SOLR-5707
> Project: Solr
> Issue Type: New Feature
>Reporter: Ryan Ernst
>Priority: Major
> Labels: pull-request-available
> Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Expressions should be available for use in Solr.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[ https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17950068#comment-17950068 ] David Smiley commented on SOLR-5707: bq. I don't know if i should be happy or sad that an 11 year old patch still works with barely any changes Hehe... that's a thought-provoking question. _ In this case_, I think it showed how well that Solr's pluggable abstractions have stood the test of time. Solr has shined here! ValueSourceParser for-the-win. I agree 100% with your effort to steer this change from a FieldType to ValueSourceParser, which makes way more sense. > Lucene Expressions in Solr > -- > > Key: SOLR-5707 > URL: https://issues.apache.org/jira/browse/SOLR-5707 > Project: Solr > Issue Type: New Feature >Reporter: Ryan Ernst >Priority: Major > Labels: pull-request-available > Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Expressions should be available for use in Solr. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[
https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17950057#comment-17950057
]
Chris M. Hostetter commented on SOLR-5707:
--
{quote}the PR here is really your work with barely any changes to speak of.
{quote}
I don't know if i should be happy or sad that an 11 year old patch still works
with barely any changes
{quote}CHANGES.txt should mention you first, then Ryan, then me & Kevin.
{quote}
I'll take your word for it ... I'm well past the point in life of caring about
being first author, especially for work i don't even remember doing. : )
> Lucene Expressions in Solr
> --
>
> Key: SOLR-5707
> URL: https://issues.apache.org/jira/browse/SOLR-5707
> Project: Solr
> Issue Type: New Feature
>Reporter: Ryan Ernst
>Priority: Major
> Labels: pull-request-available
> Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Expressions should be available for use in Solr.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[ https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17949868#comment-17949868 ] David Smiley commented on SOLR-5707: [~hossman] the PR here is really your work with barely any changes to speak of. This is a fantastic feature! The only thing not working is use in sort expressions when the score is referenced, which doesn't have a clean fix until Solr 10 when we're ready to switch to Lucene DoubleValuesSource which supercedes ValueSource (TODO there). You may want to follow the PR and offer any feedback on its mergeable state. CHANGES.txt should mention you first, then Ryan, then me & Kevin. > Lucene Expressions in Solr > -- > > Key: SOLR-5707 > URL: https://issues.apache.org/jira/browse/SOLR-5707 > Project: Solr > Issue Type: New Feature >Reporter: Ryan Ernst >Priority: Major > Labels: pull-request-available > Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Expressions should be available for use in Solr. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[
https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17948765#comment-17948765
]
David Smiley commented on SOLR-5707:
I'm very interested in adding this to Solr. The overall objective is to make
it easy to define simple expressions on the Solr side without hand-rolling Java
code (e.g. a custom DocTransformer or ValueSourceParser).
The DocTransformer (AKA for pseudo-fields) side is perhaps non-obvious but it
could allow processing text/string data to compute a number from it instead of
returning that whole value to the client and the client doing something with
it. Example: {{body_txt.toLowerCase().includes($query.toLowerCase())}}
yielding a boolean as 0 or 1. There's even a performance/scalability argument
for doing this, especially as distributed-search (sharding) is considered for
text fields, avoiding moving lots of data (disclaimer: just a hypothesis).
FWIW a colleague expressed a strong preference that some of our logic be done
on the Solr side even if the client could do similar logic with the returned
text – separation of concerns for relevancy oriented features.
> Lucene Expressions in Solr
> --
>
> Key: SOLR-5707
> URL: https://issues.apache.org/jira/browse/SOLR-5707
> Project: Solr
> Issue Type: New Feature
>Reporter: Ryan Ernst
>Priority: Major
> Labels: pull-request-available
> Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Expressions should be available for use in Solr.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[
https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17684917#comment-17684917
]
Kevin Risden commented on SOLR-5707:
>From https://github.com/risdenk/lucene-jmh I added Lucene Javascript compiler
>without needing SOLR-5707 yet.
{code:java}
Benchmark Mode CntScore
Error Units
BenchmarkConstantFunctions.testBasicMath1 thrpt 25 118.011
± 8.295 ops/s
BenchmarkConstantFunctions.testBasicMath2 thrpt 25 37.937
± 0.988 ops/s
BenchmarkConstantFunctions.testBasicMath3 thrpt 25 93.490
± 3.509 ops/s
BenchmarkConstantFunctions.testBasicNoMath1thrpt 25 142.449
± 7.410 ops/s
BenchmarkConstantFunctions.testBasicNoMath2thrpt 25 50.868
± 3.303 ops/s
BenchmarkConstantFunctions.testBasicNoMath3thrpt 25 157.506
± 13.974 ops/s
BenchmarkConstantFunctions.testBasicLuceneCompiler1 thrpt 25 90.476 ±
10.714 ops/s
BenchmarkConstantFunctions.testBasicLuceneCompiler2 thrpt 25 64.993 ±
11.400 ops/s
BenchmarkConstantFunctions.testBasicLuceneCompiler3 thrpt 25 109.659 ±
9.749 ops/s
{code}
for some VERY simple constant functions - the Lucene compiler doesn't
outperform...
However for some simple functions that don't access Lucene fields, the Lucene
compiler stuff does perform better:
{code:java}
Benchmark Mode CntScoreError
Units
BenchmarkSimpleFunctions.testBasicLuceneCompiler1 thrpt 25 507.238 ± 3.996
ops/s
BenchmarkSimpleFunctions.testBasicQuery1 thrpt 25 254.121 ± 4.365
ops/s
BenchmarkSimpleFunctions.testBasicSimpleQuery1 thrpt 25 349.014 ± 13.325
ops/s
{code}
thats running just regular functions - ~1.5-2x faster. So the slowness seems
like it might be actually accessing the Lucene field data to do the functions.
> Lucene Expressions in Solr
> --
>
> Key: SOLR-5707
> URL: https://issues.apache.org/jira/browse/SOLR-5707
> Project: Solr
> Issue Type: New Feature
>Reporter: Ryan Ernst
>Priority: Major
> Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Expressions should be available for use in Solr.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-5707) Lucene Expressions in Solr
[ https://issues.apache.org/jira/browse/SOLR-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648679#comment-17648679 ] Kevin Risden commented on SOLR-5707: For what it is worth - I'm looking at this related to some function query performance things. I put together a draft PR based on [^SOLR-5707_vsp.patch] - https://github.com/apache/solr/pull/1244 > Lucene Expressions in Solr > -- > > Key: SOLR-5707 > URL: https://issues.apache.org/jira/browse/SOLR-5707 > Project: Solr > Issue Type: New Feature >Reporter: Ryan Ernst >Priority: Major > Attachments: SOLR-5707.patch, SOLR-5707_vsp.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Expressions should be available for use in Solr. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
