[jira] [Updated] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields and text fields
[ https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove updated SOLR-10292: --- Attachment: SOLR-10292.patch Tests added and passing. This does not add any additional evaluators. I think those can be added in other tickets. All evaluators are supported by this stream so anything you think to add (regex matching, sentence creation, etc...) will work. The stream works with both multi-valued and sing-valued fields in so much that it will treat single-valued fields as a collection with a single item. > Add cartesian Streaming Expression to build cartesian products from > multi-value fields and text fields > -- > > Key: SOLR-10292 > URL: https://issues.apache.org/jira/browse/SOLR-10292 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > Attachments: SOLR-10292.patch, SOLR-10292.patch, SOLR-10292.patch, > SOLR-10292.patch > > > Currently all the Streaming Expression such as rollups, intersections, fetch > etc, work on single value fields. The *cartesian* expression would create a > stream of tuples from a single tuple with a multi-value field. This would > allow multi-valued fields to be operated on by the wider library of Streaming > Expression. > For example a single tuple with a multi-valued field: > id: 1 > author: [Jim, Jack, Steve] > Would be transformed in the following three tuples: > id:1 > author:Jim > id:1 > author:Jack > id:1 > author:Steve -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields and text fields
[ https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove updated SOLR-10292: --- Attachment: SOLR-10292.patch Fixes a missed case in FieldEvaluator. Now this turns both object arrays and Iterables which are not lists into ArrayLists. > Add cartesian Streaming Expression to build cartesian products from > multi-value fields and text fields > -- > > Key: SOLR-10292 > URL: https://issues.apache.org/jira/browse/SOLR-10292 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > Attachments: SOLR-10292.patch, SOLR-10292.patch, SOLR-10292.patch > > > Currently all the Streaming Expression such as rollups, intersections, fetch > etc, work on single value fields. The *cartesian* expression would create a > stream of tuples from a single tuple with a multi-value field. This would > allow multi-valued fields to be operated on by the wider library of Streaming > Expression. > For example a single tuple with a multi-valued field: > id: 1 > author: [Jim, Jack, Steve] > Would be transformed in the following three tuples: > id:1 > author:Jim > id:1 > author:Jack > id:1 > author:Steve -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields and text fields
[ https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove updated SOLR-10292: --- Attachment: SOLR-10292.patch Includes an implementation of read(). Also, updates FieldEvaluator to support getting multi-valued fields from the tuple. For simplicity in all evaluators, if FieldEvaluator finds an Object[] (object array) it will convert that into an ArrayList (preserving value order). This allows us to only have to check for Collection in evaluators or streams and not have to worry about object arrays. I don't have any tests yet so I'm crossing my fingers the logic is playing out as I expect it to. > Add cartesian Streaming Expression to build cartesian products from > multi-value fields and text fields > -- > > Key: SOLR-10292 > URL: https://issues.apache.org/jira/browse/SOLR-10292 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > Attachments: SOLR-10292.patch, SOLR-10292.patch > > > Currently all the Streaming Expression such as rollups, intersections, fetch > etc, work on single value fields. The *cartesian* expression would create a > stream of tuples from a single tuple with a multi-value field. This would > allow multi-valued fields to be operated on by the wider library of Streaming > Expression. > For example a single tuple with a multi-valued field: > id: 1 > author: [Jim, Jack, Steve] > Would be transformed in the following three tuples: > id:1 > author:Jim > id:1 > author:Jack > id:1 > author:Steve -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields and text fields
[
https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930824#comment-15930824
]
Dennis Gove commented on SOLR-10292:
My concern with relying on sort() is that it requires reading all tuples before
doing the sort. If we're just providing a way to order the generated tuples I
think using a sort() stream would be too costly.
I agree that the result of each evaluator will be based on the original tuple.
But if multiple evaluators, e1 and e2, are used then the resulting tuples will
look like
{code}
{
fieldA : e1[0],
fieldB : e2[0],
},
{
fieldA : e1[0],
fieldB : e2[1],
},
{
fieldA : e1[0],
fieldB : e2[2],
},
{
fieldA : e1[1],
fieldB : e2[0],
},
{
fieldA : e1[1],
fieldB : e2[1],
},
{
fieldA : e1[1],
fieldB : e2[2],
},
{code}
> Add cartesian Streaming Expression to build cartesian products from
> multi-value fields and text fields
> --
>
> Key: SOLR-10292
> URL: https://issues.apache.org/jira/browse/SOLR-10292
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
> Attachments: SOLR-10292.patch
>
>
> Currently all the Streaming Expression such as rollups, intersections, fetch
> etc, work on single value fields. The *cartesian* expression would create a
> stream of tuples from a single tuple with a multi-value field. This would
> allow multi-valued fields to be operated on by the wider library of Streaming
> Expression.
> For example a single tuple with a multi-valued field:
> id: 1
> author: [Jim, Jack, Steve]
> Would be transformed in the following three tuples:
> id:1
> author:Jim
> id:1
> author:Jack
> id:1
> author:Steve
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields and text fields
[ https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930354#comment-15930354 ] Dennis Gove commented on SOLR-10292: Also, in search of a better parameter name than "productSort". I want it to imply that this is only creating a sort on the new tuples and does not resort the entire stream. > Add cartesian Streaming Expression to build cartesian products from > multi-value fields and text fields > -- > > Key: SOLR-10292 > URL: https://issues.apache.org/jira/browse/SOLR-10292 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > Attachments: SOLR-10292.patch > > > Currently all the Streaming Expression such as rollups, intersections, fetch > etc, work on single value fields. The *cartesian* expression would create a > stream of tuples from a single tuple with a multi-value field. This would > allow multi-valued fields to be operated on by the wider library of Streaming > Expression. > For example a single tuple with a multi-valued field: > id: 1 > author: [Jim, Jack, Steve] > Would be transformed in the following three tuples: > id:1 > author:Jim > id:1 > author:Jack > id:1 > author:Steve -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields and text fields
[
https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-10292:
---
Attachment: SOLR-10292.patch
Implements everything except the read() function.
Expression is
{code}
cartesian(
,
[as newName],
[as newName],
[productSort="how to order new tuples"
)
{code}
1. Will create a tuple for each value in the field, and return in the order the
values appear in the field
{code}
cartesian(
,
multivaluedField
)
{code}
2. Will create a tuple for each value in the field, and return in the order of
the ascending order of the values in the field
{code}
cartesian(
,
multivaluedField,
productSort="multivaluedField ASC"
)
{code}
3. Will create a tuple for each value in the evaluated expression, putting the
value in the same fieldName, and return new tuples in ascending order of the
evaluated values
{code}
cartesian(
,
sentence(fieldA) as fieldA,
productSort="fieldA ASC"
)
{code}
4. Will create a tuple for each value in evaluated regex and sentence
{code}
cartesian(
,
sentence(fieldA) as newField,
regexGroups(fieldB, "some regex expression generating groups") as fieldB
productSort="fieldB ASC, newField DESC"
)
{code}
> Add cartesian Streaming Expression to build cartesian products from
> multi-value fields and text fields
> --
>
> Key: SOLR-10292
> URL: https://issues.apache.org/jira/browse/SOLR-10292
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
> Attachments: SOLR-10292.patch
>
>
> Currently all the Streaming Expression such as rollups, intersections, fetch
> etc, work on single value fields. The *cartesian* expression would create a
> stream of tuples from a single tuple with a multi-value field. This would
> allow multi-valued fields to be operated on by the wider library of Streaming
> Expression.
> For example a single tuple with a multi-valued field:
> id: 1
> author: [Jim, Jack, Steve]
> Would be transformed in the following three tuples:
> id:1
> author:Jim
> id:1
> author:Jack
> id:1
> author:Steve
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10303) Add date/time Stream Evaluators
[
https://issues.apache.org/jira/browse/SOLR-10303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929963#comment-15929963
]
Dennis Gove commented on SOLR-10303:
Can we call this a DatePartEvaluator?
This seems to only work on field values (via the field name). There'll need to
be support for acting on the results of other evaluators.
> Add date/time Stream Evaluators
> ---
>
> Key: SOLR-10303
> URL: https://issues.apache.org/jira/browse/SOLR-10303
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>
> This ticket will add Stream Evaluators that extract date/time values from a
> Solr date field. The following Evaluators will be supported:
> hour (date)
> minute (date)
> month (date)
> monthname(date)
> quarter(date)
> second (date)
> year(date)
> Syntax:
> {code}
> select(id,
>year(recdate) as year,
>month(recdate) as month,
>day(recdate) as day,
>search(logs, q="blah", fl="id, recdate", sort="recdate asc",
> qt="/export"))
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields and text fields
[ https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929953#comment-15929953 ] Dennis Gove commented on SOLR-10292: I like this. Also, a new set of evaluator types which return arrays/lists. > Add cartesian Streaming Expression to build cartesian products from > multi-value fields and text fields > -- > > Key: SOLR-10292 > URL: https://issues.apache.org/jira/browse/SOLR-10292 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > > Currently all the Streaming Expression such as rollups, intersections, fetch > etc, work on single value fields. The *cartesian* expression would create a > stream of tuples from a single tuple with a multi-value field. This would > allow multi-valued fields to be operated on by the wider library of Streaming > Expression. > For example a single tuple with a multi-valued field: > id: 1 > author: [Jim, Jack, Steve] > Would be transformed in the following three tuples: > id:1 > author:Jim > id:1 > author:Jack > id:1 > author:Steve -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields
[
https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929350#comment-15929350
]
Dennis Gove edited comment on SOLR-10292 at 3/17/17 2:36 AM:
-
I'm not a huge fan of the function name, but feature-wise I think this would be
good.
{code}
cartesian(
,
by="field1,field2",
sort="field1 ASC"
)
{code}
1. Supports any incoming stream
2. Allows you to do the product over multiple fields
3. Allows you to indicate a sort order for the new tuples, if not provided will
default to incoming order of values in by fields
4. If a non-array exists in the by fields then this will just return that
single tuple, no need to error out. Allows for fields where a mixture of multi
and single valued exist.
Anything I'm missing?
was (Author: dpgove):
I'm not a huge fan of the function name, but feature-wise I think this would be
good.
{code}
cartesian(
,
by="field1,field2",
sort="field1 ASC"
)
{code}
1. Supports any incoming stream
2. Allows you to do the product over multiple fields
3. Allows you to indicate a sort order for the new tuples, if not provided will
default to incoming order of values in by fields
Anything I'm missing?
> Add cartesian Streaming Expression to build cartesian products from
> multi-value fields
> --
>
> Key: SOLR-10292
> URL: https://issues.apache.org/jira/browse/SOLR-10292
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>
> Currently all the Streaming Expression such as rollups, intersections, fetch
> etc, work on single value fields. The *cartesian* expression would create a
> stream of tuples from a single tuple with a multi-value field. This would
> allow multi-valued fields to be operated on by the wider library of Streaming
> Expression.
> For example a single tuple with a multi-valued field:
> id: 1
> author: [Jim, Jack, Steve]
> Would be transformed in the following three tuples:
> id:1
> author:Jim
> id:1
> author:Jack
> id:1
> author:Steve
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10292) Add cartesian Streaming Expression to build cartesian products from multi-value fields
[
https://issues.apache.org/jira/browse/SOLR-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929350#comment-15929350
]
Dennis Gove commented on SOLR-10292:
I'm not a huge fan of the function name, but feature-wise I think this would be
good.
{code}
cartesian(
,
by="field1,field2",
sort="field1 ASC"
)
{code}
1. Supports any incoming stream
2. Allows you to do the product over multiple fields
3. Allows you to indicate a sort order for the new tuples, if not provided will
default to incoming order of values in by fields
Anything I'm missing?
> Add cartesian Streaming Expression to build cartesian products from
> multi-value fields
> --
>
> Key: SOLR-10292
> URL: https://issues.apache.org/jira/browse/SOLR-10292
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>
> Currently all the Streaming Expression such as rollups, intersections, fetch
> etc, work on single value fields. The *cartesian* expression would create a
> stream of tuples from a single tuple with a multi-value field. This would
> allow multi-valued fields to be operated on by the wider library of Streaming
> Expression.
> For example a single tuple with a multi-valued field:
> id: 1
> author: [Jim, Jack, Steve]
> Would be transformed in the following three tuples:
> id:1
> author:Jim
> id:1
> author:Jack
> id:1
> author:Steve
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10293) Ship with Streaming Expression examples
[ https://issues.apache.org/jira/browse/SOLR-10293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15929325#comment-15929325 ] Dennis Gove commented on SOLR-10293: I'm happy to put some together. Where would these types of things go? > Ship with Streaming Expression examples > --- > > Key: SOLR-10293 > URL: https://issues.apache.org/jira/browse/SOLR-10293 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > > In would be great if Solr shipped with Streaming Expression examples so that > users could come up to speed quickly on the basics. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10217) Add a query for the background set to the significantTerms streaming expression
[ https://issues.apache.org/jira/browse/SOLR-10217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15903656#comment-15903656 ] Dennis Gove commented on SOLR-10217: What's the reasoning behind having SignificantTermsStream accept a query instead of an incoming stream? If it accepted an incoming stream then the source data could be the result of any other stream whereas with a query the source data can only be the result of a Solr query. > Add a query for the background set to the significantTerms streaming > expression > --- > > Key: SOLR-10217 > URL: https://issues.apache.org/jira/browse/SOLR-10217 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Gethin James > Attachments: SOLR-20217.patch > > > Following the work on SOLR-10156 we now have a significantTerms expression. > Currently, the background set is always the full index. It would be great if > we could use a query to define the background set. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-8185) Add operations support to streaming metrics
[
https://issues.apache.org/jira/browse/SOLR-8185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove closed SOLR-8185.
-
Resolution: Won't Fix
This work is superceded by SOLR-9916.
> Add operations support to streaming metrics
> ---
>
> Key: SOLR-8185
> URL: https://issues.apache.org/jira/browse/SOLR-8185
> Project: Solr
> Issue Type: Improvement
> Components: SolrJ
> Reporter: Dennis Gove
>Assignee: Dennis Gove
>Priority: Minor
> Attachments: SOLR-8185.patch
>
>
> Adds support for operations on stream metrics.
> With this feature one can modify tuple values before applying to the computed
> metric. There are a lot of use-cases I can see with this - I'll describe one
> here.
> Imagine you have a RollupStream which is computing the average over some
> field but you cannot be sure that all documents have a value for that field,
> ie the value is null. When the value is null you want to treat it as a 0.
> With this feature you can accomplish that like this
> {code}
> rollup(
> search(collection1, q=*:*, fl=\"a_s,a_i,a_f\", sort=\"a_s asc\"),
> over=\"a_s\",
> avg(a_i, replace(null, withValue=0)),
> count(*),
> )
> {code}
> The operations are applied to the tuple for each metric in the stream which
> means you perform different operations on different metrics without being
> impacted by operations on other metrics.
> Adding to our previous example, imagine you want to also get the min of a
> field but do not consider null values.
> {code}
> rollup(
> search(collection1, q=*:*, fl=\"a_s,a_i,a_f\", sort=\"a_s asc\"),
> over=\"a_s\",
> avg(a_i, replace(null, withValue=0)),
> min(a_i),
> count(*),
> )
> {code}
> Also, the tuple is not modified for streams that might wrap this one. Ie, the
> only thing that sees the applied operation is that particular metric. If you
> want to apply operations for wrapping streams you can still achieve that with
> the SelectStream (SOLR-7669).
> One feature I'm investigating but this patch DOES NOT add is the ability to
> assign names to the resulting metric value. For example, to allow for
> something like this
> {code}
> rollup(
> search(collection1, q=*:*, fl=\"a_s,a_i,a_f\", sort=\"a_s asc\"),
> over=\"a_s\",
> avg(a_i, replace(null, withValue=0), as="avg_a_i_null_as_0"),
> avg(a_i),
> count(*, as="totalCount"),
> )
> {code}
> Right now that isn't possible because the identifier for each metric would be
> the same "avg_a_i" and as such both couldn't be returned. It's relatively
> easy to add but I have to investigate its impact on the SQL and FacetStream
> areas.
> Depends on SOLR-7669 (SelectStream)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8185) Add operations support to streaming metrics
[
https://issues.apache.org/jira/browse/SOLR-8185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898133#comment-15898133
]
Dennis Gove commented on SOLR-8185:
---
I agree - Evaluators take care of this feature.
> Add operations support to streaming metrics
> ---
>
> Key: SOLR-8185
> URL: https://issues.apache.org/jira/browse/SOLR-8185
> Project: Solr
> Issue Type: Improvement
> Components: SolrJ
> Reporter: Dennis Gove
>Assignee: Dennis Gove
>Priority: Minor
> Attachments: SOLR-8185.patch
>
>
> Adds support for operations on stream metrics.
> With this feature one can modify tuple values before applying to the computed
> metric. There are a lot of use-cases I can see with this - I'll describe one
> here.
> Imagine you have a RollupStream which is computing the average over some
> field but you cannot be sure that all documents have a value for that field,
> ie the value is null. When the value is null you want to treat it as a 0.
> With this feature you can accomplish that like this
> {code}
> rollup(
> search(collection1, q=*:*, fl=\"a_s,a_i,a_f\", sort=\"a_s asc\"),
> over=\"a_s\",
> avg(a_i, replace(null, withValue=0)),
> count(*),
> )
> {code}
> The operations are applied to the tuple for each metric in the stream which
> means you perform different operations on different metrics without being
> impacted by operations on other metrics.
> Adding to our previous example, imagine you want to also get the min of a
> field but do not consider null values.
> {code}
> rollup(
> search(collection1, q=*:*, fl=\"a_s,a_i,a_f\", sort=\"a_s asc\"),
> over=\"a_s\",
> avg(a_i, replace(null, withValue=0)),
> min(a_i),
> count(*),
> )
> {code}
> Also, the tuple is not modified for streams that might wrap this one. Ie, the
> only thing that sees the applied operation is that particular metric. If you
> want to apply operations for wrapping streams you can still achieve that with
> the SelectStream (SOLR-7669).
> One feature I'm investigating but this patch DOES NOT add is the ability to
> assign names to the resulting metric value. For example, to allow for
> something like this
> {code}
> rollup(
> search(collection1, q=*:*, fl=\"a_s,a_i,a_f\", sort=\"a_s asc\"),
> over=\"a_s\",
> avg(a_i, replace(null, withValue=0), as="avg_a_i_null_as_0"),
> avg(a_i),
> count(*, as="totalCount"),
> )
> {code}
> Right now that isn't possible because the identifier for each metric would be
> the same "avg_a_i" and as such both couldn't be returned. It's relatively
> easy to add but I have to investigate its impact on the SQL and FacetStream
> areas.
> Depends on SOLR-7669 (SelectStream)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8706) Add SMSStream to support SMS messaging
[ https://issues.apache.org/jira/browse/SOLR-8706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15896453#comment-15896453 ] Dennis Gove commented on SOLR-8706: --- I'm happy to house this as a submodule in https://github.com/dennisgove/streaming-expressions. > Add SMSStream to support SMS messaging > -- > > Key: SOLR-8706 > URL: https://issues.apache.org/jira/browse/SOLR-8706 > Project: Solr > Issue Type: New Feature >Reporter: Joel Bernstein > > With the TopicStream (SOLR-8588) wrapping up it makes sense to implement an > SMS Stream that wraps the TopicStream and pushes SMS messages. > Research needs to be done to determine the right SMS library. More details to > follow on syntax as the this ticket develops. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10086) Add Streaming Expression for Kafka Streams
[
https://issues.apache.org/jira/browse/SOLR-10086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15896451#comment-15896451
]
Dennis Gove commented on SOLR-10086:
For the other parameters, I'm thinking of just key/value strings. The reason
being, the usage above allows the values to be calculated out of
each tuple, like {code}topic=if(gt(a,b),raw("topic1"),raw("topic2")){code}
would allow a tuple to be sent to topic1 if a > b, else go to topic2. This type
of thing doesn't really apply for the publisher options because those are
determined at the beginning and don't change from tuple to tuple.
> Add Streaming Expression for Kafka Streams
> --
>
> Key: SOLR-10086
> URL: https://issues.apache.org/jira/browse/SOLR-10086
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Reporter: Susheel Kumar
>Priority: Minor
>
> This is being asked to have SolrCloud pull data from Kafka topic periodically
> using DataImport Handler.
> Adding streaming expression support to pull data from Kafka would be good
> feature to have.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-10086) Add Streaming Expression for Kafka Streams
[
https://issues.apache.org/jira/browse/SOLR-10086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15896023#comment-15896023
]
Dennis Gove edited comment on SOLR-10086 at 3/5/17 2:01 AM:
I'm developing this under a different project, so I dunno if the conversation
should occur here or elsewhere, but to keep everyone in the loop, here's what
I'm thinking about for the expression for a Kafka Producer.
{code}
kafkaProducer(
,
topic=,
bootstrapServers=,
key=, // optional, if not provided then no key used when sending
record
keyType=[string,long,double,boolean], // only required if using key and can't
figure out from key param (knowing long or double is tricky)
value= // optional, if not provided then whole tuple as json
string is used
valueType=[string,long,double,boolean], // only required if can't figure out
from value param (knowing long or double is tricky)
partition=, // optional, if not provided then no partition used
when sending record
)
{code}
Anything that is is allowed to be any valid StreamEvaluator, such as
{code}
raw("foo") // raw value "foo"
"foo" // value of field foo
add(foo, bar) // value of foo + value of bar
{code}
There are other parameters allowed with a Kafka v0.10.x producer, such as
{code}
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
{code}
These could be handled like the parameters in a Search Stream, where any
parameters not explicitly known are just passed along. We could do that here so
people can set whatever parameters they want on the publisher and we don't have
to keep updating our supported list as the Kafka lib version changes.
was (Author: dpgove):
I'm developing this under a different project, so I dunno if the conversation
should occur here or elsewhere, but to keep everyone in the loop, here's what
I'm thinking about for the expression for a Kafka Producer.
{code}
kafkaProducer(
,
topic=,
bootstrapServers=,
key=, // optional, if not provided then no key used when sending
record
keyType=[string,long,double,boolean], // only required if using key and can't
figure out from key param (knowing long or double is tricky)
value= // optional, if not provided then whole tuple as json
string is used
valueType=[string,long,double,boolean], // only required if can't figure out
from value param (knowing long or double is tricky)
partition=, // optional, if not provided then no partition used
when sending record
)
{code}
Anything that is is allowed to be any valid StreamEvaluator, such as
{code}
raw("foo") // raw value "foo"
"foo" // value of field foo
add(foo, bar) // value of foo + value of bar
{code}
There are other parameters allowed with a Kafka v0.10.x producer, such as
{code}
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
{code}
These could be handled like the parameters in a Search Stream, where any
parameters no explicitly known are just passed along. We could do that here so
people can set whatever parameters they want on the publisher and we don't have
to keep updating our supported list as the Kafka lib version changes.
> Add Streaming Expression for Kafka Streams
> --
>
> Key: SOLR-10086
> URL: https://issues.apache.org/jira/browse/SOLR-10086
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Reporter: Susheel Kumar
>Priority: Minor
>
> This is being asked to have SolrCloud pull data from Kafka topic periodically
> using DataImport Handler.
> Adding streaming expression support to pull data from Kafka would be good
> feature to have.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10086) Add Streaming Expression for Kafka Streams
[
https://issues.apache.org/jira/browse/SOLR-10086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15896023#comment-15896023
]
Dennis Gove commented on SOLR-10086:
I'm developing this under a different project, so I dunno if the conversation
should occur here or elsewhere, but to keep everyone in the loop, here's what
I'm thinking about for the expression for a Kafka Producer.
{code}
kafkaProducer(
,
topic=,
bootstrapServers=,
key=, // optional, if not provided then no key used when sending
record
keyType=[string,long,double,boolean], // only required if using key and can't
figure out from key param (knowing long or double is tricky)
value= // optional, if not provided then whole tuple as json
string is used
valueType=[string,long,double,boolean], // only required if can't figure out
from value param (knowing long or double is tricky)
partition=, // optional, if not provided then no partition used
when sending record
)
{code}
Anything that is is allowed to be any valid StreamEvaluator, such as
{code}
raw("foo") // raw value "foo"
"foo" // value of field foo
add(foo, bar) // value of foo + value of bar
{code}
There are other parameters allowed with a Kafka v0.10.x producer, such as
{code}
props.put("acks", "all");
props.put("retries", 0);
props.put("batch.size", 16384);
props.put("linger.ms", 1);
props.put("buffer.memory", 33554432);
{code}
These could be handled like the parameters in a Search Stream, where any
parameters no explicitly known are just passed along. We could do that here so
people can set whatever parameters they want on the publisher and we don't have
to keep updating our supported list as the Kafka lib version changes.
> Add Streaming Expression for Kafka Streams
> --
>
> Key: SOLR-10086
> URL: https://issues.apache.org/jira/browse/SOLR-10086
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Reporter: Susheel Kumar
>Priority: Minor
>
> This is being asked to have SolrCloud pull data from Kafka topic periodically
> using DataImport Handler.
> Adding streaming expression support to pull data from Kafka would be good
> feature to have.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10086) Add Streaming Expression for Kafka Streams
[ https://issues.apache.org/jira/browse/SOLR-10086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15895515#comment-15895515 ] Dennis Gove commented on SOLR-10086: For anyone interested, I've created https://github.com/dennisgove/streaming-expressions which is setup as a suite of projects that will have various streaming components which shouldn't be added to solr core. > Add Streaming Expression for Kafka Streams > -- > > Key: SOLR-10086 > URL: https://issues.apache.org/jira/browse/SOLR-10086 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Reporter: Susheel Kumar >Priority: Minor > > This is being asked to have SolrCloud pull data from Kafka topic periodically > using DataImport Handler. > Adding streaming expression support to pull data from Kafka would be good > feature to have. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove resolved SOLR-9916. --- Resolution: Fixed Fix Version/s: 6.5 > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > Assignee: Dennis Gove > Fix For: 6.5 > > Attachments: SOLR-9916.patch, SOLR-9916.patch, SOLR-9916.patch, > SOLR-9916-precommit.patch > > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9916) Add arithmetic operations to the SelectStream
[
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-9916:
--
Attachment: SOLR-9916-precommit.patch
This is the patch that fixes the {code}ant precommit{code} failure.
> Add arithmetic operations to the SelectStream
> -
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
> Assignee: Dennis Gove
> Attachments: SOLR-9916.patch, SOLR-9916.patch, SOLR-9916.patch,
> SOLR-9916-precommit.patch
>
>
> One of the things that will be needed as the SQL implementation matures is
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best
> place to add this functionality.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15862846#comment-15862846 ] Dennis Gove commented on SOLR-9916: --- Looks like I forgot to add the package-info.java file to the new package. Will add. > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein >Assignee: Dennis Gove > Attachments: SOLR-9916.patch, SOLR-9916.patch, SOLR-9916.patch > > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15862546#comment-15862546 ] Dennis Gove commented on SOLR-9916: --- I've added descriptions of each evaluator to Trunk Changes at https://cwiki.apache.org/confluence/display/solr/Internal+-+Trunk+Changes+to+Document. I'll move these to the true location after 6.4 has been cut. > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein >Assignee: Dennis Gove > Attachments: SOLR-9916.patch, SOLR-9916.patch, SOLR-9916.patch > > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9759) Admin UI should post streaming expressions
[
https://issues.apache.org/jira/browse/SOLR-9759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861953#comment-15861953
]
Dennis Gove commented on SOLR-9759:
---
Just to be clear, the expression you attempted to execute was larger than
8192bytes?
> Admin UI should post streaming expressions
> --
>
> Key: SOLR-9759
> URL: https://issues.apache.org/jira/browse/SOLR-9759
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: UI
>Affects Versions: 6.2.1
>Reporter: Gus Heck
>
> Haven't had the chance to test this in 6.3, but in 6.2.1 I just ran into
> request entity too large, when I pasted an expression into the admin ui to
> begin debugging it...
> Furthermore, the UI gives no indication of any error at all, leading one to
> sit, waiting for the response. Firefox JavaScript console shows a 413
> response and this:
> {code}
> 11:01:11.095 Error: JSON.parse: unexpected character at line 1 column 1 of
> the JSON data
> $scope.doStream/<@http://localhost:8984/solr/js/angular/controllers/stream.js:48:24
> v/http://localhost:8984/solr/libs/angular-resource.min.js:33:133
> processQueue@http://localhost:8984/solr/libs/angular.js:13193:27
> scheduleProcessQueue/<@http://localhost:8984/solr/libs/angular.js:13209:27
> $RootScopeProvider/this.$gethttp://localhost:8984/solr/libs/angular.js:14406:16
> $RootScopeProvider/this.$gethttp://localhost:8984/solr/libs/angular.js:14222:15
> $RootScopeProvider/this.$gethttp://localhost:8984/solr/libs/angular.js:14511:13
> done@http://localhost:8984/solr/libs/angular.js:9669:36
> completeRequest@http://localhost:8984/solr/libs/angular.js:9859:7
> requestLoaded@http://localhost:8984/solr/libs/angular.js:9800:9
> 1angular.js:11617:18
> consoleLog/<()angular.js:11617
> $ExceptionHandlerProvider/this.$get processQueue()angular.js:13201
> scheduleProcessQueue/<()angular.js:13209
> $RootScopeProvider/this.$get $RootScopeProvider/this.$get $RootScopeProvider/this.$get done()angular.js:9669
> completeRequest()angular.js:9859
> requestLoaded()angular.js:9800
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9962) need to extend classes in org.apache.solr.client.solrj.io.stream.metrics package
[ https://issues.apache.org/jira/browse/SOLR-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15861946#comment-15861946 ] Dennis Gove commented on SOLR-9962: --- In what ways do you want to extend them? What do they not do that you need to do? > need to extend classes in org.apache.solr.client.solrj.io.stream.metrics > package > > > Key: SOLR-9962 > URL: https://issues.apache.org/jira/browse/SOLR-9962 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Server >Affects Versions: 6.3 >Reporter: radhakrishnan devarajan >Priority: Trivial > Fix For: 6.4 > > > i want to extend the update(Tuple tuple) method in MaxMetric,. MinMetric, > SumMetric, MeanMetric classes. > can you please make the below metioned variables and methods in the above > mentioned classes as protected so that it will be easy to extend > variables > --- > longMax > doubleMax > columnName > and > methods > --- > init -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Reopened] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove reopened SOLR-9916: --- Assignee: Dennis Gove Mistake closing this. I intend to keep it open until I update the wiki docs. > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > Assignee: Dennis Gove > Attachments: SOLR-9916.patch, SOLR-9916.patch, SOLR-9916.patch > > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove closed SOLR-9916. - Resolution: Fixed > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > Attachments: SOLR-9916.patch, SOLR-9916.patch, SOLR-9916.patch > > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove updated SOLR-9916: -- Attachment: SOLR-9916.patch Final patch. All tests pass. Applies to both branch_6x and master. > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > Attachments: SOLR-9916.patch, SOLR-9916.patch, SOLR-9916.patch > > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860117#comment-15860117 ] Dennis Gove commented on SOLR-9916: --- [~joel.bernstein], just wondering on your thoughts on this? I completely removed the Operations that were added last summer and replaced them with these Evaluators. > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > Attachments: SOLR-9916.patch, SOLR-9916.patch > > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9916) Add arithmetic operations to the SelectStream
[
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-9916:
--
Attachment: SOLR-9916.patch
This is complete. All tests pass.
I have deleted and replaced all existing BooleanOperations with their requisite
BooleanEvaluators and added additional evaluators.
The registration of default evaluators looks like this
{code}
// Stream Evaluators
.withFunctionName("val", RawValueEvaluator.class)
// Boolean Stream Evaluators
.withFunctionName("and", AndEvaluator.class)
.withFunctionName("eor", ExclusiveOrEvaluator.class)
.withFunctionName("eq", EqualsEvaluator.class)
.withFunctionName("gt", GreaterThanEvaluator.class)
.withFunctionName("gteq", GreaterThanEqualToEvaluator.class)
.withFunctionName("lt", LessThanEvaluator.class)
.withFunctionName("lteq", LessThanEqualToEvaluator.class)
.withFunctionName("not", NotEvaluator.class)
.withFunctionName("or", OrEvaluator.class)
// Number Stream Evaluators
.withFunctionName("abs", AbsoluteValueEvaluator.class)
.withFunctionName("add", AddEvaluator.class)
.withFunctionName("div", DivideEvaluator.class)
.withFunctionName("mult", MultiplyEvaluator.class)
.withFunctionName("sub", SubtractEvaluator.class)
// Conditional Stream Evaluators
.withFunctionName("if", IfThenElseEvaluator.class)
{code}
All evaluators accept the following parameter formats
{code}
add(abc,def) // field abc + field def
add(sub(abc,def),ghi) // (field abc - field def) + field ghi
add(abc,9) // field abc + 9
add(sum(abc), def) // field sum(abc) + field def
{code}
Basically, when an evaluator is parsing its parameters it will first determine
if the parameter is another evaluator. If not, then it will determine if the
parameter is a Double, Long, or Boolean raw value. If not, it will treat the
parameter as a field name. This allows us to use field names like "sum(abc)"
which are the result of rollups and use raw values and use embedded evaluators.
> Add arithmetic operations to the SelectStream
> -
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
> Attachments: SOLR-9916.patch, SOLR-9916.patch
>
>
> One of the things that will be needed as the SQL implementation matures is
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best
> place to add this functionality.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855882#comment-15855882 ] Dennis Gove commented on SOLR-9916: --- Future work for another ticket will be to remove the use of BooleanOperation over to use evaluators. > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > Attachments: SOLR-9916.patch > > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9916) Add arithmetic operations to the SelectStream
[
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-9916:
--
Attachment: SOLR-9916.patch
Includes the following evaluators
{code}
StreamEvaluator
ComplexEvaluator // evaluators allowing other StreamEvaluators as parameters
(looking for better class name)
NumberEvaluator // evaluators resulting in a Number return value
AbsoluteValueEvaluator // abs(a)
AddEvaluator // add(a,b,...,z)
DivideEvaluator // div(a,b)
MultiplyEvaluator // mult(a,b,...,z)
SubtractEvaluator // sub(a,b)
BooleanEvaluator // evaluators resulting in a Boolean return value
AndEvaluator // and(a,b,...,z) == true iff all all true
EqualsEvaluator // eq(a,b,...,z) == true iff all are equal
GreaterThanEqualToEvaluator
GreaterThanEvaluator
LessThanEqualToEvaluator
LessThanEvaluator
OrEvaluator
ConditionalEvaluator // evaluators performing a conditional and returning
an Object based on the result
IfThenElseEvaluator
SimpleEvaluator // evaluators not allowing other StreamEvaluators as
parameters (looking for a better class name)
FieldEvaluator // return the value of the field - not something that needs
to be expressed in the expression
{code}
Still needed:
* ValueEvaluator
* Addition Testing
* Handling of null raw value
* Additional evaluators
> Add arithmetic operations to the SelectStream
> -
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
> Attachments: SOLR-9916.patch
>
>
> One of the things that will be needed as the SQL implementation matures is
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best
> place to add this functionality.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9916) Add arithmetic operations to the SelectStream
[
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15855876#comment-15855876
]
Dennis Gove commented on SOLR-9916:
---
I've decided to go in a slightly different direction with this. After
refamilarizing myself with StreamOperations, it became clear that operations
are meant to modify tuples. For example, the ReplaceOperation replaces some
field value with some other value via the {code} void operate(Tuple){code}
function. Newer operations like the BooleanOperation extends that with an
evaluate() function, but I find it's not quite as clear as it could be.
Bringing this back to the problem we want to solve: we want to evaluate some
value based on a tuple. This isn't meant to modify a tuple but instead to
calculate new values from other values that exist within tuple. This is true
whether we are adding, multiplying, determining equality, greater than, or
choosing with an if/else. We are evaluating, but not modifying, the tuple.
To solve this problem, I'm introducing a new set of classes called
StreamEvaluators. StreamEvaluators follow the same functional expression
structure as everything else within the streaming sphere and define the
function {code}public Object evaluate(Tuple){code}. The object returned from
this function is the result of the evaluation against the tuple. For example,
the result returned for the expresssion {code}add(a,b){code} is the result of
field a added to field b. The datatype of the returned result is determined by
the evaluator and the source field types. For example, {code}add(a,b){code}
could reasonably return a Number, either Long or Double, while
{code}eq(a,b){code} could reasonably return a Boolean, while
{code}if(eq(a,b),c,d){code} could reasonably return any type.
StreamEvaluators come in two basic flavors - those that can contain other
evaluators and those that can't.
{code}
add(a,b) // field a + field b
sub(add(a,b),c) // (a + b) - c
mult(if(gt("a",b),a,b),c) // if field a > field b then a * c else b * c
if(eq(a,b),val(34),c) // if a == b then value 34 else c
if(eq(a,b),val(foo),c) // if a == b then value "foo" else c
if(eq(a,null),b,c) // if a is null then b else c
{code}
There are a couple pieces of note in here.
* null is a special case and will be treated as a standard null value
* A ValueEvaluator {code}val(), val(), val(){code}
will evaluate to the raw value contained within
** This allows us to easily distinguish between field names and raw string
values.
* Within any other evaluator, a string, quoted or not, will be considered a
field name
As a result of this, the class structure is turning into this.
{code}
StreamEvaluator
ComplexEvaluator // evaluators allowing other StreamEvaluators as parameters
(looking for better class name)
NumberEvaluator // evaluators resulting in a Number return value
AbsoluteValueEvaluator // abs(a)
AddEvaluator // add(a,b,...,z)
DivideEvaluator // div(a,b)
MultiplyEvaluator // mult(a,b,...,z)
SubtractEvaluator // sub(a,b)
BooleanEvaluator // evaluators resulting in a Boolean return value
AndEvaluator // and(a,b,...,z) == true iff all all true
EqualsEvaluator // eq(a,b,...,z) == true iff all are equal
GreaterThanEqualToEvaluator
GreaterThanEvaluator
LessThanEqualToEvaluator
LessThanEvaluator
OrEvaluator
ConditionalEvaluator // evaluators performing a conditional and returning
an Object based on the result
IfThenElseEvaluator
SimpleEvaluator // evaluators not allowing other StreamEvaluators as
parameters (looking for a better class name)
ValueEvaluator // return the raw value as-is
FieldEvaluator // return the value of the field - not something that needs
to be expressed in the expression
{code}
StreamEvaluators will become a type of parameter supported by the SelectStream
and executed *after* the execution of operations in that select clause. The
result of the evaluation will be put into the tuple under the 'as' field name.
{code}select(...,add(a,b) as aPlusb){code}
> Add arithmetic operations to the SelectStream
> -
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to
> support these types of operations.
> It looks like adding
[jira] [Commented] (SOLR-10087) StreamHandler should be able to use runtimeLib jars
[
https://issues.apache.org/jira/browse/SOLR-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849228#comment-15849228
]
Dennis Gove commented on SOLR-10087:
I think this is a good addition.
> StreamHandler should be able to use runtimeLib jars
> ---
>
> Key: SOLR-10087
> URL: https://issues.apache.org/jira/browse/SOLR-10087
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Minor
> Attachments: SOLR-10087.patch
>
>
> StreamHandler currently can't uses jars that via the runtimeLib and Blob
> Store api. This is because the StreamHandler uses core.getResourceLoader()
> instead of core.getMemClassLoader() for loading classes.
> An example of this working with the fix is here:
> https://github.com/risdenk/solr_custom_streaming_expressions
> Steps:
> {code}
> # Inspired by
> https://cwiki.apache.org/confluence/display/solr/Adding+Custom+Plugins+in+SolrCloud+Mode
> # Start Solr with enabling Blob Store
> ./bin/solr start -c -f -a "-Denable.runtime.lib=true"
> # Create test collection
> ./bin/solr create -c test
> # Create .system collection
> curl 'http://localhost:8983/solr/admin/collections?action=CREATE&name=.system'
> # Build custom streaming expression jar
> (cd custom-streaming-expression && mvn clean package)
> # Upload jar to .system collection using Blob Store API
> (https://cwiki.apache.org/confluence/display/solr/Blob+Store+API)
> curl -X POST -H 'Content-Type: application/octet-stream' --data-binary
> @custom-streaming-expression/target/custom-streaming-expression-1.0-SNAPSHOT.jar
> 'http://localhost:8983/solr/.system/blob/test'
> # List all blobs that are stored
> curl 'http://localhost:8983/solr/.system/blob?omitHeader=true'
> # Add the jar to the runtime lib
> curl 'http://localhost:8983/solr/test/config' -H
> 'Content-type:application/json' -d '{
>"add-runtimelib": { "name":"test", "version":1 }
> }'
> # Create custom streaming expression using work from SOLR-9103
> # Patch from SOLR-10087 is required for StreamHandler to load the runtimeLib
> jar
> curl 'http://localhost:8983/solr/test/config' -H
> 'Content-type:application/json' -d '{
> "create-expressible": {
> "name": "customstreamingexpression",
> "class": "com.test.solr.CustomStreamingExpression",
> "runtimeLib": true
> }
> }'
> # Test the custom streaming expression
> curl 'http://localhost:8983/solr/test/stream?expr=customstreamingexpression()'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10087) StreamHandler should be able to use runtimeLib jars
[ https://issues.apache.org/jira/browse/SOLR-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848918#comment-15848918 ] Dennis Gove commented on SOLR-10087: I'm hesitant about the ability to post binaries into Solr - I feel like it's a huge security concern. That said, this feature isn't adding the ability to post binaries, just the ability to make use of posted binaries in streaming. If the ability to post binaries is something that is already considered safe and/or the benefits outweigh the drawbacks, then I don't see any reason why streaming shouldn't support it. > StreamHandler should be able to use runtimeLib jars > --- > > Key: SOLR-10087 > URL: https://issues.apache.org/jira/browse/SOLR-10087 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Minor > Attachments: SOLR-10087.patch > > > StreamHandler currently can't uses jars that via the runtimeLib and Blob > Store api. This is because the StreamHandler uses core.getResourceLoader() > instead of core.getMemClassLoader() for loading classes. > An example of this working with the fix is here: > https://github.com/risdenk/solr_custom_streaming_expressions -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10085) Streaming Expressions result-set fields not in order
[
https://issues.apache.org/jira/browse/SOLR-10085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15848571#comment-15848571
]
Dennis Gove commented on SOLR-10085:
There's no guarantee of field order in the tuple. Fields can be added to the
tuple at any point in the stream (consider a select after a rollup after a
facet after a search). There's no way to know that the user wants a particular
order of the fields in the tuple.
Beyond that, order of fields in json is not guaranteed. So while the stream
handler could try to re-order the fields in the tuple, there is no guarantee
that the serializer (or the client's deserializer) will honor that.
bq. From RFC 7159 -The JavaScript Object Notation (JSON) Data Interchange
Format: An object is an unordered collection of zero or more name/value pairs,
where a name is a string and a value is a string, number, boolean, null,
object, or array.
> Streaming Expressions result-set fields not in order
>
>
> Key: SOLR-10085
> URL: https://issues.apache.org/jira/browse/SOLR-10085
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: faceting
>Affects Versions: 6.3
> Environment: Windows 8.1, Java 8
>Reporter: Yeo Zheng Lin
> Labels: json, streaming
>
> I'm trying out the Streaming Expressions in Solr 6.3.0.
> Currently, I'm facing the issue of not being able to get the fields in the
> result-set to be displayed in the same order as what I put in the query.
> For example, when I execute this query:
> http://localhost:8983/solr/collection1/stream?expr=facet(collection1,
> q="*:*",
> buckets="id,cost,quantity",
> bucketSorts="cost desc",
> bucketSizeLimit=100,
> sum(cost),
> sum(quantity),
> min(cost),
> min(quantity),
> max(cost),
> max(quantity),
> avg(cost),
> avg(quantity),
> count(*))&indent=true
> I get the following in the result-set.
>{
> "result-set":{"docs":[
> {
> "min(quantity)":12.21,
> "avg(quantity)":12.21,
> "sum(cost)":256.33,
> "max(cost)":256.33,
> "count(*)":1,
> "min(cost)":256.33,
> "cost":256.33,
> "avg(cost)":256.33,
> "quantity":12.21,
> "id":"01",
> "sum(quantity)":12.21,
> "max(quantity)":12.21},
> {
> "EOF":true,
> "RESPONSE_TIME":359}]}}
> The fields are displayed randomly all over the place, instead of the order
> sum, min, max, avg as in the query. This may cause confusion to user who look
> at the output. Possible improvement to display the fields in the result-set
> in the same order as the query
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15803185#comment-15803185 ] Dennis Gove edited comment on SOLR-9916 at 1/6/17 1:45 AM: --- Of course. sum is a metric. was (Author: dpgove): Oh jeez. Of course. sum is a metric. > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15803185#comment-15803185 ] Dennis Gove commented on SOLR-9916: --- Oh jeez. Of course. sum is a metric. > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9916) Add arithmetic operations to the SelectStream
[
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15803159#comment-15803159
]
Dennis Gove commented on SOLR-9916:
---
Sounds good.
What is {code}sum(a_i){code}? Is that calculating the sum over a multivalued
field? (if so...didn't know we were supporting multivalued fields, really cool)
> Add arithmetic operations to the SelectStream
> -
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best
> place to add this functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9916) Add arithmetic operations to the SelectStream
[
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15803113#comment-15803113
]
Dennis Gove commented on SOLR-9916:
---
Looking at the current state of Operations, the following class structure exists
{code}
StreamOperation
ConcatOperation
BooleanOperation
AndOperation
LeafOperation
EqualsOperation
GreaterThanEqualToOperation
GreaterThanOperation
LessThanEqualToOperation
LessThanOperation
NotOperation
OrOperation
ReduceOperation
DistinctOperation
GroupOperation
ReplaceOperation (and associated hidden ReplaceWithFieldOperation,
ReplaceWithValueOperation)
{code}
I'd like to enhance this slightly to the following
{code}
StreamOperation
BooleanOperation
AndOperation
LeafOperation
EqualsOperation
GreaterThanEqualToOperation
GreaterThanOperation
LessThanEqualToOperation
LessThanOperation
NotOperation
OrOperation
ComparisonOperation
IfOperation
ModificationOperation
AbsoluteValueOperation
AdditionOperation
ConcatOperation
DivisionOperation
ModuloOperation
MultiplicationOperation
ReplaceOperation (and associated hidden ReplaceWithFieldOperation,
ReplaceWithValueOperation)
SubtractionOperation
ReduceOperation
DistinctOperation
GroupOperation
{code}
This will allow us to support arbitrarily complex operations in the Select
stream. It accomplishes this in 3 ways.
h3. Comparison Operation
First, add an if/then/else concept with the ComparisonOperation. Embedded
operations will be supported, either Modification or Comparison.
The full supported structure is
{code}
if(boolean, field | modification | comparison, field | modification |
comparison)
{code}
For example,
{code}
if(boolean(...), fieldA, fieldB)
ex: if(gt(a,b), a, b) // if a > b then a else b
if(boolean(...), modification(...), modification)
ex: if(gt(a,b), sub(a,b), sub(b,a)) // if a > b then a - b else b - a
if(boolean(...), comparison(...), comparison(...))
ex: if(gt(a,b), if(or(c,d), a, b), if(and(c,d), a, b)) // if a > b then (if c
or d then a else b) else (if c and d then a else b)
{code}
h3. ModificationOperations with Embedded Operations
Second, enhance ModificationOperations to support embedded operations, either
Modification or Comparison.
The full supported structure is
{code}
modification(field | modification | comparison [, field | modification |
comparison])
{code}
For example,
{code}
modification(fieldA [,fieldB])
ex: add(a,b) // a + b
modification(fieldA [,modification(...)]) // order doesn't matter
ex: add(a, div(b,c)) // a + (b/c)
add(div(b,c), a) // (b/c) + a
modification(fieldA [,comparison(...)]) // order doesn't matter
ex: add(a, if(gt(b,c),b,c)) // if b > c then a + b else a + c
add(if(gt(b,c),b,c), a) // if b > c then a + b else a + c
{code}
h3. BooleanOperations with Embedded Operations
Third, enhance BooleanOperations to support embedded operations, either
Modification or Comparison. Each would support the following constructs
The full supported structure is
{code}
boolean(field | modification | comparison [, field | modification | comparison])
{code}
{code}
boolean(fieldA [,fieldB])
ex: gt(a,b)
boolean(fieldA [,modification(...)]) // order doesn't matter
ex: gt(a, add(b,c)) // is a > (b + c)
gt(add(b,c), a) // is (b + c) > a
boolean(fieldA [,comparison(...)]) // order doesn't matter
ex: gt(a, if(gt(b,c),b,c)) // if b > c then is a > b else is a > c
gt(if(gt(b,c),b,c), a) // if b > c then is b > a else is c > a
{code}
[~joel.bernstein], I'm interested in your thoughts on this.
> Add arithmetic operations to the SelectStream
> -
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best
> place to add this functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9916) Add arithmetic operations to the SelectStream
[
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802935#comment-15802935
]
Dennis Gove edited comment on SOLR-9916 at 1/6/17 12:01 AM:
I'm going to start implementing these as Operations. I'll be sure to support
the cases of operations within operations like
{code}
plus(div(a,replace(b,null,0)),c)
{code}
was (Author: dpgove):
I'm going to start implementing these as Operations.
> Add arithmetic operations to the SelectStream
> -
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best
> place to add this functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802935#comment-15802935 ] Dennis Gove edited comment on SOLR-9916 at 1/5/17 11:59 PM: I'm going to start implementing these as Operations. was (Author: dpgove): I'mm going to start implementing these as Operations. > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9916) Add arithmetic operations to the SelectStream
[
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802934#comment-15802934
]
Dennis Gove edited comment on SOLR-9916 at 1/5/17 11:59 PM:
I think this is a good idea.
Select already supports an "as" concept, so something like would be possible
already
{code}
select(plus(a,b) as outfield, )
{code}
was (Author: dpgove):
I think this is a good idea.
Select already supports an "as" concept, so something like would be possible
already
{code}
select(add(a,b) as outfield, )
{code}
> Add arithmetic operations to the SelectStream
> -
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best
> place to add this functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9916) Add arithmetic operations to the SelectStream
[ https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802935#comment-15802935 ] Dennis Gove commented on SOLR-9916: --- I'mm going to start implementing these as Operations. > Add arithmetic operations to the SelectStream > - > > Key: SOLR-9916 > URL: https://issues.apache.org/jira/browse/SOLR-9916 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein > > One of the things that will be needed as the SQL implementation matures is > the ability to do arithmetic operations. For example: > select (a+b) from x; > select sum(a)+sum(b) from x; > We will need to support arithmetic operations within the Streaming API to > support these types of operations. > It looks like adding arithmetic operations to the SelectStream is the best > place to add this functionality. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9916) Add arithmetic operations to the SelectStream
[
https://issues.apache.org/jira/browse/SOLR-9916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802934#comment-15802934
]
Dennis Gove commented on SOLR-9916:
---
I think this is a good idea.
Select already supports an "as" concept, so something like would be possible
already
{code}
select(add(a,b) as outfield, )
{code}
> Add arithmetic operations to the SelectStream
> -
>
> Key: SOLR-9916
> URL: https://issues.apache.org/jira/browse/SOLR-9916
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>
> One of the things that will be needed as the SQL implementation matures is
> the ability to do arithmetic operations. For example:
> select (a+b) from x;
> select sum(a)+sum(b) from x;
> We will need to support arithmetic operations within the Streaming API to
> support these types of operations.
> It looks like adding arithmetic operations to the SelectStream is the best
> place to add this functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-7495) Unexpected docvalues type NUMERIC when grouping by a int facet
[
https://issues.apache.org/jira/browse/SOLR-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove closed SOLR-7495.
-
Resolution: Fixed
Fix Version/s: 6.4
> Unexpected docvalues type NUMERIC when grouping by a int facet
> --
>
> Key: SOLR-7495
> URL: https://issues.apache.org/jira/browse/SOLR-7495
> Project: Solr
> Issue Type: Bug
>Affects Versions: 5.0, 5.1, 5.2, 5.3
>Reporter: Fabio Batista da Silva
> Assignee: Dennis Gove
> Fix For: 6.4
>
> Attachments: SOLR-7495.patch, SOLR-7495.patch, SOLR-7495.patch
>
>
> Hey All,
> After upgrading from solr 4.10 to 5.1 with solr could
> I'm getting a IllegalStateException when i try to facet a int field.
> IllegalStateException: unexpected docvalues type NUMERIC for field 'year'
> (expected=SORTED). Use UninvertingReader or index with docvalues.
> schema.xml
> {code}
>
>
>
>
>
>
> multiValued="false" required="true"/>
> multiValued="false" required="true"/>
>
>
> stored="true"/>
>
>
>
> />
> sortMissingLast="true"/>
> positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> precisionStep="0" positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> positionIncrementGap="100">
>
>
> words="stopwords.txt" />
>
> maxGramSize="15"/>
>
>
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>
>
> positionIncrementGap="100">
>
>
> words="stopwords.txt" />
>
> maxGramSize="15"/>
>
>
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>
>
> class="solr.SpatialRecursivePrefixTreeFieldType" geo="true"
> distErrPct="0.025" maxDistErr="0.09" units="degrees" />
>
> id
> name
>
>
> {code}
> query :
> {code}
> http://solr.dev:8983/solr/my_collection/select?wt=json&fl=id&fq=index_type:foobar&group=true&group.field=year_make_model&group.facet=true&facet=true&facet.field=year
> {code}
> Exception :
> {code}
> ull:org.apache.solr.common.SolrException: Exception during facet.field: year
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:627)
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at org.apache.solr.request.SimpleFacets$2.execute(SimpleFacets.java:566)
> at
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:637)
> at
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:280)
> at
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:106)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:222)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
>
[jira] [Assigned] (SOLR-7495) Unexpected docvalues type NUMERIC when grouping by a int facet
[
https://issues.apache.org/jira/browse/SOLR-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove reassigned SOLR-7495:
-
Assignee: Dennis Gove
> Unexpected docvalues type NUMERIC when grouping by a int facet
> --
>
> Key: SOLR-7495
> URL: https://issues.apache.org/jira/browse/SOLR-7495
> Project: Solr
> Issue Type: Bug
>Affects Versions: 5.0, 5.1, 5.2, 5.3
>Reporter: Fabio Batista da Silva
> Assignee: Dennis Gove
> Attachments: SOLR-7495.patch, SOLR-7495.patch, SOLR-7495.patch
>
>
> Hey All,
> After upgrading from solr 4.10 to 5.1 with solr could
> I'm getting a IllegalStateException when i try to facet a int field.
> IllegalStateException: unexpected docvalues type NUMERIC for field 'year'
> (expected=SORTED). Use UninvertingReader or index with docvalues.
> schema.xml
> {code}
>
>
>
>
>
>
> multiValued="false" required="true"/>
> multiValued="false" required="true"/>
>
>
> stored="true"/>
>
>
>
> />
> sortMissingLast="true"/>
> positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> precisionStep="0" positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> positionIncrementGap="100">
>
>
> words="stopwords.txt" />
>
> maxGramSize="15"/>
>
>
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>
>
> positionIncrementGap="100">
>
>
> words="stopwords.txt" />
>
> maxGramSize="15"/>
>
>
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>
>
> class="solr.SpatialRecursivePrefixTreeFieldType" geo="true"
> distErrPct="0.025" maxDistErr="0.09" units="degrees" />
>
> id
> name
>
>
> {code}
> query :
> {code}
> http://solr.dev:8983/solr/my_collection/select?wt=json&fl=id&fq=index_type:foobar&group=true&group.field=year_make_model&group.facet=true&facet=true&facet.field=year
> {code}
> Exception :
> {code}
> ull:org.apache.solr.common.SolrException: Exception during facet.field: year
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:627)
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at org.apache.solr.request.SimpleFacets$2.execute(SimpleFacets.java:566)
> at
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:637)
> at
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:280)
> at
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:106)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:222)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at
&g
[jira] [Updated] (SOLR-7495) Unexpected docvalues type NUMERIC when grouping by a int facet
[
https://issues.apache.org/jira/browse/SOLR-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-7495:
--
Attachment: SOLR-7495.patch
This patch applies cleanly to branch_6x.
I did see the test fail before applying the change to SimpleFacets and then saw
it pass after applying the change to SimpleFacets. Going to run through all the
tests before committing this change.
> Unexpected docvalues type NUMERIC when grouping by a int facet
> --
>
> Key: SOLR-7495
> URL: https://issues.apache.org/jira/browse/SOLR-7495
> Project: Solr
> Issue Type: Bug
>Affects Versions: 5.0, 5.1, 5.2, 5.3
>Reporter: Fabio Batista da Silva
> Attachments: SOLR-7495.patch, SOLR-7495.patch, SOLR-7495.patch
>
>
> Hey All,
> After upgrading from solr 4.10 to 5.1 with solr could
> I'm getting a IllegalStateException when i try to facet a int field.
> IllegalStateException: unexpected docvalues type NUMERIC for field 'year'
> (expected=SORTED). Use UninvertingReader or index with docvalues.
> schema.xml
> {code}
>
>
>
>
>
>
> multiValued="false" required="true"/>
> multiValued="false" required="true"/>
>
>
> stored="true"/>
>
>
>
> />
> sortMissingLast="true"/>
> positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> precisionStep="0" positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> positionIncrementGap="100">
>
>
> words="stopwords.txt" />
>
> maxGramSize="15"/>
>
>
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>
>
> positionIncrementGap="100">
>
>
> words="stopwords.txt" />
>
> maxGramSize="15"/>
>
>
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>
>
> class="solr.SpatialRecursivePrefixTreeFieldType" geo="true"
> distErrPct="0.025" maxDistErr="0.09" units="degrees" />
>
> id
> name
>
>
> {code}
> query :
> {code}
> http://solr.dev:8983/solr/my_collection/select?wt=json&fl=id&fq=index_type:foobar&group=true&group.field=year_make_model&group.facet=true&facet=true&facet.field=year
> {code}
> Exception :
> {code}
> ull:org.apache.solr.common.SolrException: Exception during facet.field: year
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:627)
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at org.apache.solr.request.SimpleFacets$2.execute(SimpleFacets.java:566)
> at
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:637)
> at
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:280)
> at
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:106)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:222)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455
[jira] [Issue Comment Deleted] (SOLR-7495) Unexpected docvalues type NUMERIC when grouping by a int facet
[
https://issues.apache.org/jira/browse/SOLR-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-7495:
--
Comment: was deleted
(was: I applied the patch for just SimpleFacetsTest.java to branch_6x and ran
the tests and it did not fail. Is this still an issue in 6x?)
> Unexpected docvalues type NUMERIC when grouping by a int facet
> --
>
> Key: SOLR-7495
> URL: https://issues.apache.org/jira/browse/SOLR-7495
> Project: Solr
> Issue Type: Bug
>Affects Versions: 5.0, 5.1, 5.2, 5.3
>Reporter: Fabio Batista da Silva
> Attachments: SOLR-7495.patch, SOLR-7495.patch
>
>
> Hey All,
> After upgrading from solr 4.10 to 5.1 with solr could
> I'm getting a IllegalStateException when i try to facet a int field.
> IllegalStateException: unexpected docvalues type NUMERIC for field 'year'
> (expected=SORTED). Use UninvertingReader or index with docvalues.
> schema.xml
> {code}
>
>
>
>
>
>
> multiValued="false" required="true"/>
> multiValued="false" required="true"/>
>
>
> stored="true"/>
>
>
>
> />
> sortMissingLast="true"/>
> positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> precisionStep="0" positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> positionIncrementGap="100">
>
>
> words="stopwords.txt" />
>
> maxGramSize="15"/>
>
>
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>
>
> positionIncrementGap="100">
>
>
> words="stopwords.txt" />
>
> maxGramSize="15"/>
>
>
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>
>
> class="solr.SpatialRecursivePrefixTreeFieldType" geo="true"
> distErrPct="0.025" maxDistErr="0.09" units="degrees" />
>
> id
> name
>
>
> {code}
> query :
> {code}
> http://solr.dev:8983/solr/my_collection/select?wt=json&fl=id&fq=index_type:foobar&group=true&group.field=year_make_model&group.facet=true&facet=true&facet.field=year
> {code}
> Exception :
> {code}
> ull:org.apache.solr.common.SolrException: Exception during facet.field: year
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:627)
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at org.apache.solr.request.SimpleFacets$2.execute(SimpleFacets.java:566)
> at
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:637)
> at
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:280)
> at
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:106)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:222)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
&
[jira] [Commented] (SOLR-7495) Unexpected docvalues type NUMERIC when grouping by a int facet
[
https://issues.apache.org/jira/browse/SOLR-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15799933#comment-15799933
]
Dennis Gove commented on SOLR-7495:
---
I applied the patch for just SimpleFacetsTest.java to branch_6x and ran the
tests and it did not fail. Is this still an issue in 6x?
> Unexpected docvalues type NUMERIC when grouping by a int facet
> --
>
> Key: SOLR-7495
> URL: https://issues.apache.org/jira/browse/SOLR-7495
> Project: Solr
> Issue Type: Bug
>Affects Versions: 5.0, 5.1, 5.2, 5.3
>Reporter: Fabio Batista da Silva
> Attachments: SOLR-7495.patch, SOLR-7495.patch
>
>
> Hey All,
> After upgrading from solr 4.10 to 5.1 with solr could
> I'm getting a IllegalStateException when i try to facet a int field.
> IllegalStateException: unexpected docvalues type NUMERIC for field 'year'
> (expected=SORTED). Use UninvertingReader or index with docvalues.
> schema.xml
> {code}
>
>
>
>
>
>
> multiValued="false" required="true"/>
> multiValued="false" required="true"/>
>
>
> stored="true"/>
>
>
>
> />
> sortMissingLast="true"/>
> positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> precisionStep="0" positionIncrementGap="0"/>
> positionIncrementGap="0"/>
> positionIncrementGap="100">
>
>
> words="stopwords.txt" />
>
> maxGramSize="15"/>
>
>
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>
>
> positionIncrementGap="100">
>
>
> words="stopwords.txt" />
>
> maxGramSize="15"/>
>
>
>
> words="stopwords.txt" />
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>
>
>
> class="solr.SpatialRecursivePrefixTreeFieldType" geo="true"
> distErrPct="0.025" maxDistErr="0.09" units="degrees" />
>
> id
> name
>
>
> {code}
> query :
> {code}
> http://solr.dev:8983/solr/my_collection/select?wt=json&fl=id&fq=index_type:foobar&group=true&group.field=year_make_model&group.facet=true&facet=true&facet.field=year
> {code}
> Exception :
> {code}
> ull:org.apache.solr.common.SolrException: Exception during facet.field: year
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:627)
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at org.apache.solr.request.SimpleFacets$2.execute(SimpleFacets.java:566)
> at
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:637)
> at
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:280)
> at
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:106)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:222)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[jira] [Closed] (SOLR-5616) Make grouping code use response builder needDocList
[
https://issues.apache.org/jira/browse/SOLR-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove closed SOLR-5616.
-
Resolution: Fixed
> Make grouping code use response builder needDocList
> ---
>
> Key: SOLR-5616
> URL: https://issues.apache.org/jira/browse/SOLR-5616
> Project: Solr
> Issue Type: Bug
>Reporter: Steven Bower
> Assignee: Dennis Gove
> Fix For: 6.4
>
> Attachments: SOLR-5616.patch, SOLR-5616.patch, SOLR-5616.patch
>
>
> Right now the grouping code does this to check if it needs to generate a
> docList for grouped results:
> {code}
> if (rb.doHighlights || rb.isDebug() || params.getBool(MoreLikeThisParams.MLT,
> false) ){
> ...
> }
> {code}
> this is ugly because any new component that needs a doclist, from grouped
> results, will need to modify QueryComponent to add a check to this if.
> Ideally this should just use the rb.isNeedDocList() flag...
> Coincidentally this boolean is really never used at for non-grouped results
> it always gets generated..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5616) Make grouping code use response builder needDocList
[
https://issues.apache.org/jira/browse/SOLR-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-5616:
--
Fix Version/s: 6.4
> Make grouping code use response builder needDocList
> ---
>
> Key: SOLR-5616
> URL: https://issues.apache.org/jira/browse/SOLR-5616
> Project: Solr
> Issue Type: Bug
>Reporter: Steven Bower
> Assignee: Dennis Gove
> Fix For: 6.4
>
> Attachments: SOLR-5616.patch, SOLR-5616.patch, SOLR-5616.patch
>
>
> Right now the grouping code does this to check if it needs to generate a
> docList for grouped results:
> {code}
> if (rb.doHighlights || rb.isDebug() || params.getBool(MoreLikeThisParams.MLT,
> false) ){
> ...
> }
> {code}
> this is ugly because any new component that needs a doclist, from grouped
> results, will need to modify QueryComponent to add a check to this if.
> Ideally this should just use the rb.isNeedDocList() flag...
> Coincidentally this boolean is really never used at for non-grouped results
> it always gets generated..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Reopened] (SOLR-5616) Make grouping code use response builder needDocList
[
https://issues.apache.org/jira/browse/SOLR-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove reopened SOLR-5616:
---
Assignee: Dennis Gove (was: Erick Erickson)
> Make grouping code use response builder needDocList
> ---
>
> Key: SOLR-5616
> URL: https://issues.apache.org/jira/browse/SOLR-5616
> Project: Solr
> Issue Type: Bug
>Reporter: Steven Bower
> Assignee: Dennis Gove
> Fix For: 6.4
>
> Attachments: SOLR-5616.patch, SOLR-5616.patch, SOLR-5616.patch
>
>
> Right now the grouping code does this to check if it needs to generate a
> docList for grouped results:
> {code}
> if (rb.doHighlights || rb.isDebug() || params.getBool(MoreLikeThisParams.MLT,
> false) ){
> ...
> }
> {code}
> this is ugly because any new component that needs a doclist, from grouped
> results, will need to modify QueryComponent to add a check to this if.
> Ideally this should just use the rb.isNeedDocList() flag...
> Coincidentally this boolean is really never used at for non-grouped results
> it always gets generated..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-3990) index size unavailable in gui/mbeans unless replication handler configured
[ https://issues.apache.org/jira/browse/SOLR-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove closed SOLR-3990. - Resolution: Fixed Committed into both master and branch_6x. Thank you, Shawn! > index size unavailable in gui/mbeans unless replication handler configured > -- > > Key: SOLR-3990 > URL: https://issues.apache.org/jira/browse/SOLR-3990 > Project: Solr > Issue Type: Improvement > Components: web gui >Affects Versions: 4.0 >Reporter: Shawn Heisey >Assignee: Dennis Gove >Priority: Minor > Fix For: 6.4 > > Attachments: SOLR-3990.patch, SOLR-3990.patch, SOLR-3990.patch > > > Unless you configure the replication handler, the on-disk size of each core's > index seems to be unavailable in the gui or from the mbeans handler. If you > are not doing replication, you should still be able to get the size of each > index without configuring things that won't be used. > Also, I would like to get the size of the index in a consistent unit of > measurement, probably MB. I understand the desire to give people a human > readable unit next to a number that's not enormous, but it's difficult to do > programmatic comparisons between values such as 787.33 MB and 23.56 GB. That > may mean that the number needs to be available twice, one format to be shown > in the admin GUI and both formats available from the mbeans handler, for > scripting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3990) index size unavailable in gui/mbeans unless replication handler configured
[ https://issues.apache.org/jira/browse/SOLR-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove updated SOLR-3990: -- Fix Version/s: (was: 6.0) (was: 4.10) 6.4 > index size unavailable in gui/mbeans unless replication handler configured > -- > > Key: SOLR-3990 > URL: https://issues.apache.org/jira/browse/SOLR-3990 > Project: Solr > Issue Type: Improvement > Components: web gui >Affects Versions: 4.0 >Reporter: Shawn Heisey >Assignee: Dennis Gove >Priority: Minor > Fix For: 6.4 > > Attachments: SOLR-3990.patch, SOLR-3990.patch, SOLR-3990.patch > > > Unless you configure the replication handler, the on-disk size of each core's > index seems to be unavailable in the gui or from the mbeans handler. If you > are not doing replication, you should still be able to get the size of each > index without configuring things that won't be used. > Also, I would like to get the size of the index in a consistent unit of > measurement, probably MB. I understand the desire to give people a human > readable unit next to a number that's not enormous, but it's difficult to do > programmatic comparisons between values such as 787.33 MB and 23.56 GB. That > may mean that the number needs to be available twice, one format to be shown > in the admin GUI and both formats available from the mbeans handler, for > scripting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3990) index size unavailable in gui/mbeans unless replication handler configured
[ https://issues.apache.org/jira/browse/SOLR-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove updated SOLR-3990: -- Attachment: SOLR-3990.patch Only changes in this are the hashes for the diff (applied to current master). No code changes. > index size unavailable in gui/mbeans unless replication handler configured > -- > > Key: SOLR-3990 > URL: https://issues.apache.org/jira/browse/SOLR-3990 > Project: Solr > Issue Type: Improvement > Components: web gui >Affects Versions: 4.0 >Reporter: Shawn Heisey >Assignee: Dennis Gove >Priority: Minor > Fix For: 4.10, 6.0 > > Attachments: SOLR-3990.patch, SOLR-3990.patch, SOLR-3990.patch > > > Unless you configure the replication handler, the on-disk size of each core's > index seems to be unavailable in the gui or from the mbeans handler. If you > are not doing replication, you should still be able to get the size of each > index without configuring things that won't be used. > Also, I would like to get the size of the index in a consistent unit of > measurement, probably MB. I understand the desire to give people a human > readable unit next to a number that's not enormous, but it's difficult to do > programmatic comparisons between values such as 787.33 MB and 23.56 GB. That > may mean that the number needs to be available twice, one format to be shown > in the admin GUI and both formats available from the mbeans handler, for > scripting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3990) index size unavailable in gui/mbeans unless replication handler configured
[ https://issues.apache.org/jira/browse/SOLR-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15798792#comment-15798792 ] Dennis Gove commented on SOLR-3990: --- I agree, I think it still makes sense to put this in a more central location. I'd expect to be able to ask the core what the index size is. > index size unavailable in gui/mbeans unless replication handler configured > -- > > Key: SOLR-3990 > URL: https://issues.apache.org/jira/browse/SOLR-3990 > Project: Solr > Issue Type: Improvement > Components: web gui >Affects Versions: 4.0 >Reporter: Shawn Heisey >Assignee: Dennis Gove >Priority: Minor > Fix For: 4.10, 6.0 > > Attachments: SOLR-3990.patch, SOLR-3990.patch > > > Unless you configure the replication handler, the on-disk size of each core's > index seems to be unavailable in the gui or from the mbeans handler. If you > are not doing replication, you should still be able to get the size of each > index without configuring things that won't be used. > Also, I would like to get the size of the index in a consistent unit of > measurement, probably MB. I understand the desire to give people a human > readable unit next to a number that's not enormous, but it's difficult to do > programmatic comparisons between values such as 787.33 MB and 23.56 GB. That > may mean that the number needs to be available twice, one format to be shown > in the admin GUI and both formats available from the mbeans handler, for > scripting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-3990) index size unavailable in gui/mbeans unless replication handler configured
[ https://issues.apache.org/jira/browse/SOLR-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove reassigned SOLR-3990: - Assignee: Dennis Gove (was: Shawn Heisey) > index size unavailable in gui/mbeans unless replication handler configured > -- > > Key: SOLR-3990 > URL: https://issues.apache.org/jira/browse/SOLR-3990 > Project: Solr > Issue Type: Improvement > Components: web gui >Affects Versions: 4.0 >Reporter: Shawn Heisey >Assignee: Dennis Gove >Priority: Minor > Fix For: 4.10, 6.0 > > Attachments: SOLR-3990.patch, SOLR-3990.patch > > > Unless you configure the replication handler, the on-disk size of each core's > index seems to be unavailable in the gui or from the mbeans handler. If you > are not doing replication, you should still be able to get the size of each > index without configuring things that won't be used. > Also, I would like to get the size of the index in a consistent unit of > measurement, probably MB. I understand the desire to give people a human > readable unit next to a number that's not enormous, but it's difficult to do > programmatic comparisons between values such as 787.33 MB and 23.56 GB. That > may mean that the number needs to be available twice, one format to be shown > in the admin GUI and both formats available from the mbeans handler, for > scripting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Welcome Christine Poerschke to the PMC
Wahoo! Congratulations Christine! On Fri, Dec 30, 2016 at 7:46 AM, Adrien Grand wrote: > I am pleased to announce that Christine Poerschke has accepted the PMC's > invitation to join. > > Welcome Christine! > > Adrien >
Re: MemoryIndex and query not matching a Long value
I've found the problem. It appears the Analyzer I'm using is not
appropriate for numeric values. This results in a Query object of the wrong
type. Whereas when querying point values the type PointRangeQuery should be
used but a TermQuery is created instead. If I directly create a
PointRangeQuery with LongPoint.newExactQuery() then the expected
results are found.
// creates wrong Query object
Analyzer analyzer = new StandardAnalyzer();
QueryParser parser = new QueryParser(null, analyzer);
queries.add(parser.parse("a_i:1"));
Now I need to figure out how I can turn an arbitrary lucene query (against
unknown field type) into a valid Query object for the field being queried
against. I think it'd be Ok to, in this case, determine the type of Query
based on the type of the search value...but I don't yet know if there's an
Analyzer that will do that, quasi-magically.
- Dennis
On Mon, Jan 2, 2017 at 5:01 PM, Dennis Gove wrote:
> Thank you. This isn't a situation where I have access to a schema so I
> don't think I can make use of the FieldType methods. I'm implementing a
> Match stream as part of the streaming api (our discussion in
> https://issues.apache.org/jira/browse/SOLR-8530).
>
> An arbitrary tuple can come in with calculated values so the types of the
> values can't necessarily be determined from a schema. Due to this, I'm
> taking all fields in the tuple and constructing a document (see
> https://github.com/dennisgove/lucene-solr/blob/
> SolrMatch/solr/core/src/java/org/apache/solr/handler/
> LuceneMatchStream.java#L195).
>
> (side note: the working name is LuceneMatchStream because atm it only
> accepts Lucene syntax for the queries)
>
> - Dennis
>
>
> On Mon, Jan 2, 2017 at 4:45 PM, David Smiley
> wrote:
>
>> LongPoint uses the Points API. If you are using a Solr QParserPlugin,
>> it's not going to use that API. Assuming you're in Solr land, I think you
>> should be using utility methods on FieldType (lookup from schema) which can
>> create the Field instances to be put on the document.
>>
>> ~ David
>>
>> > On Jan 2, 2017, at 4:33 PM, Dennis Gove wrote:
>> >
>> > I'm messing around with a MemoryIndex and am indexing a field of type
>> Long. From everything I can tell, this should be added into a Document as
>> type org.apache.lucene.document.LongPoint. However, when I try to match
>> it with a query of form "a_i:1" it doesn't match.
>> >
>> > For example, full document is
>> > {
>> > a_s:"hello1",
>> > a_i:1
>> > }
>> > with Query object created from
>> > "a_i:1"
>> >
>> > the return in call to
>> > index.search(query)
>> >
>> > is 0 (ie, a non-match)
>> >
>> > The only thing I can think of is that the document field should
>> actually be something else, or that the creation of a Query object from
>> "a_i:1" isn't going to match a LongPoint value.
>> >
>> > Thanks!
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>
Re: MemoryIndex and query not matching a Long value
Thank you. This isn't a situation where I have access to a schema so I
don't think I can make use of the FieldType methods. I'm implementing a
Match stream as part of the streaming api (our discussion in
https://issues.apache.org/jira/browse/SOLR-8530).
An arbitrary tuple can come in with calculated values so the types of the
values can't necessarily be determined from a schema. Due to this, I'm
taking all fields in the tuple and constructing a document (see
https://github.com/dennisgove/lucene-solr/blob/SolrMatch/solr/core/src/java/org/apache/solr/handler/LuceneMatchStream.java#L195
).
(side note: the working name is LuceneMatchStream because atm it only
accepts Lucene syntax for the queries)
- Dennis
On Mon, Jan 2, 2017 at 4:45 PM, David Smiley
wrote:
> LongPoint uses the Points API. If you are using a Solr QParserPlugin,
> it's not going to use that API. Assuming you're in Solr land, I think you
> should be using utility methods on FieldType (lookup from schema) which can
> create the Field instances to be put on the document.
>
> ~ David
>
> > On Jan 2, 2017, at 4:33 PM, Dennis Gove wrote:
> >
> > I'm messing around with a MemoryIndex and am indexing a field of type
> Long. From everything I can tell, this should be added into a Document as
> type org.apache.lucene.document.LongPoint. However, when I try to match
> it with a query of form "a_i:1" it doesn't match.
> >
> > For example, full document is
> > {
> > a_s:"hello1",
> > a_i:1
> > }
> > with Query object created from
> > "a_i:1"
> >
> > the return in call to
> > index.search(query)
> >
> > is 0 (ie, a non-match)
> >
> > The only thing I can think of is that the document field should actually
> be something else, or that the creation of a Query object from "a_i:1"
> isn't going to match a LongPoint value.
> >
> > Thanks!
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
MemoryIndex and query not matching a Long value
I'm messing around with a MemoryIndex and am indexing a field of type Long.
>From everything I can tell, this should be added into a Document as type
org.apache.lucene.document.LongPoint. However, when I try to match it with
a query of form "a_i:1" it doesn't match.
For example, full document is
{
a_s:"hello1",
a_i:1
}
with Query object created from
"a_i:1"
the return in call to
index.search(query)
is 0 (ie, a non-match)
The only thing I can think of is that the document field should actually be
something else, or that the creation of a Query object from "a_i:1" isn't
going to match a LongPoint value.
Thanks!
[jira] [Commented] (SOLR-8530) Add HavingStream to Streaming API and StreamingExpressions
[
https://issues.apache.org/jira/browse/SOLR-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15793092#comment-15793092
]
Dennis Gove commented on SOLR-8530:
---
One problem I ran into when I was approaching the Match (or SolrMatch, I like
David's idea about naming) implementation was that the classes needed for
in-memory index don't exist in the SolrJ library. This means it would create a
dependency on something outside SolrJ. If I remember correctly, the specific
pieces I was trying to implement was the parsing of a Solr query to a Lucene
compatible query. This is because the in-memory index requires Lucene syntax
while I wanted the SolrMatch to accept Solr syntax.
> Add HavingStream to Streaming API and StreamingExpressions
> --
>
> Key: SOLR-8530
> URL: https://issues.apache.org/jira/browse/SOLR-8530
> Project: Solr
> Issue Type: Improvement
> Components: SolrJ
>Affects Versions: 6.0
>Reporter: Dennis Gove
>Priority: Minor
>
> The goal here is to support something similar to SQL's HAVING clause where
> one can filter documents based on data that is not available in the index.
> For example, filter the output of a reduce() based on the calculated
> metrics.
> {code}
> having(
> reduce(
> search(.),
> sum(cost),
> on=customerId
> ),
> q="sum(cost):[500 TO *]"
> )
> {code}
> This example would return all where the total spent by each distinct customer
> is >= 500. The total spent is calculated via the sum(cost) metric in the
> reduce stream.
> The intent is to support as the filters in the having(...) clause the full
> query syntax of a search(...) clause. I see this being possible in one of two
> ways.
> 1. Use Lucene's MemoryIndex and as each tuple is read out of the underlying
> stream creating an instance of MemoryIndex and apply the query to it. If the
> result of that is >0 then the tuple should be returned from the HavingStream.
> 2. Create an in-memory solr index via something like RamDirectory, read all
> tuples into that in-memory index using the UpdateStream, and then stream out
> of that all the matching tuples from the query.
> There are benefits to each approach but I think the easiest and most direct
> one is the MemoryIndex approach. With MemoryIndex it isn't necessary to read
> all incoming tuples before returning a single tuple. With a MemoryIndex there
> is a need to parse the solr query parameters and create a valid Lucene query
> but I suspect that can be done using existing QParser implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9720) Refactor responsewriter to remove dependencies on TupleStream etc
[ https://issues.apache.org/jira/browse/SOLR-9720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634453#comment-15634453 ] Dennis Gove commented on SOLR-9720: --- The changes wrt Explanation appear fine. There isn't any test coverage of the toMap(...) function so that could be added on a future date. > Refactor responsewriter to remove dependencies on TupleStream etc > - > > Key: SOLR-9720 > URL: https://issues.apache.org/jira/browse/SOLR-9720 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Noble Paul > Attachments: SOLR-9720.patch, SOLR-9720.patch, SOLR-9720.patch, > SOLR_9720_fix_JSONWriterTest.patch > > > ResponseWriters are designed to be agnostic of components and they should > work with the well know set of types we already support -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9661) Explain of select that uses replace() throws exception
[
https://issues.apache.org/jira/browse/SOLR-9661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625245#comment-15625245
]
Dennis Gove commented on SOLR-9661:
---
I can't think of a clever, generic way to handle this case.
{code}ReplaceWithFieldOperation{code}
and
{code}ReplaceWithValueOperation{code}
are both aliased by the
{code}
ReplaceOperation
{code}
class.
Within either of those classes there's no simple way to know which function
name is assigned. There are a couple of ways to do it but they feel rather
hokey. I'm going to give it some thought before deciding on a course of action.
> Explain of select that uses replace() throws exception
> --
>
> Key: SOLR-9661
> URL: https://issues.apache.org/jira/browse/SOLR-9661
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Gus Heck
>
> {code}
> select(
>search(test, q="table:article ", fl="edge_id", sort="edge_id desc",
> rows=10),
>edge_id,
>replace(type,null, withValue="1")
> )
> {code}
> as a streaming expression produced this stack trace:
> {code}
> ERROR (qtp1989972246-17) [c:test s:shard1 r:core_node1
> x:test_shard1_replica1] o.a.s.s.HttpSolrCall null:java.io.IOException: Unable
> to find function name for class
> 'org.apache.solr.client.solrj.io.ops.ReplaceWithValueOperation'
> at
> org.apache.solr.client.solrj.io.stream.expr.StreamFactory.getFunctionName(StreamFactory.java:335)
> at
> org.apache.solr.client.solrj.io.ops.ReplaceWithValueOperation.toExpression(ReplaceWithValueOperation.java:108)
> at
> org.apache.solr.client.solrj.io.ops.ReplaceOperation.toExpression(ReplaceOperation.java:81)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExpression(SelectStream.java:148)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExplanation(SelectStream.java:164)
> at
> org.apache.solr.client.solrj.io.stream.PushBackStream.toExplanation(PushBackStream.java:56)
> at
> org.apache.solr.client.solrj.io.stream.ComplementStream.toExplanation(ComplementStream.java:132)
> at
> org.apache.solr.client.solrj.io.stream.PushBackStream.toExplanation(PushBackStream.java:56)
> at
> org.apache.solr.client.solrj.io.stream.ComplementStream.toExplanation(ComplementStream.java:132)
> at
> org.apache.solr.client.solrj.io.stream.RankStream.toExplanation(RankStream.java:142)
> at
> org.apache.solr.client.solrj.io.stream.PushBackStream.toExplanation(PushBackStream.java:56)
> at
> org.apache.solr.client.solrj.io.stream.MergeStream.toExplanation(MergeStream.java:136)
> at
> org.apache.solr.client.solrj.io.stream.HashJoinStream.toExplanation(HashJoinStream.java:174)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExplanation(SelectStream.java:159)
> at
> org.apache.solr.client.solrj.io.stream.RankStream.toExplanation(RankStream.java:142)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExplanation(SelectStream.java:159)
> at
> org.apache.solr.handler.StreamHandler.handleRequestBody(StreamHandler.java:205)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.Co
[jira] [Commented] (SOLR-9661) Explain of select that uses replace() throws exception
[
https://issues.apache.org/jira/browse/SOLR-9661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15625220#comment-15625220
]
Dennis Gove commented on SOLR-9661:
---
I've thrown together a simple test that shows this exception.
{code}
@Test
public void replaceToExplanation() throws Exception{
StreamFactory factory = new StreamFactory().withFunctionName("replace",
ReplaceOperation.class);
StreamOperation operation = new ReplaceOperation("fieldA",
StreamExpressionParser.parse("replace(null, withValue=foo)"), factory);
StreamExpressionParameter expression = operation.toExpression(factory);
Explanation explanation = operation.toExplanation(factory);
}
{code}
Obviously, the toExplanation(...) line throws an exception but so does the
toExpression(...) line. I'm not sure why this hasn't come up before as
toExpression is used extensively, particularly by the parallel stream.
> Explain of select that uses replace() throws exception
> --
>
> Key: SOLR-9661
> URL: https://issues.apache.org/jira/browse/SOLR-9661
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Gus Heck
>
> {code}
> select(
>search(test, q="table:article ", fl="edge_id", sort="edge_id desc",
> rows=10),
>edge_id,
>replace(type,null, withValue="1")
> )
> {code}
> as a streaming expression produced this stack trace:
> {code}
> ERROR (qtp1989972246-17) [c:test s:shard1 r:core_node1
> x:test_shard1_replica1] o.a.s.s.HttpSolrCall null:java.io.IOException: Unable
> to find function name for class
> 'org.apache.solr.client.solrj.io.ops.ReplaceWithValueOperation'
> at
> org.apache.solr.client.solrj.io.stream.expr.StreamFactory.getFunctionName(StreamFactory.java:335)
> at
> org.apache.solr.client.solrj.io.ops.ReplaceWithValueOperation.toExpression(ReplaceWithValueOperation.java:108)
> at
> org.apache.solr.client.solrj.io.ops.ReplaceOperation.toExpression(ReplaceOperation.java:81)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExpression(SelectStream.java:148)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExplanation(SelectStream.java:164)
> at
> org.apache.solr.client.solrj.io.stream.PushBackStream.toExplanation(PushBackStream.java:56)
> at
> org.apache.solr.client.solrj.io.stream.ComplementStream.toExplanation(ComplementStream.java:132)
> at
> org.apache.solr.client.solrj.io.stream.PushBackStream.toExplanation(PushBackStream.java:56)
> at
> org.apache.solr.client.solrj.io.stream.ComplementStream.toExplanation(ComplementStream.java:132)
> at
> org.apache.solr.client.solrj.io.stream.RankStream.toExplanation(RankStream.java:142)
> at
> org.apache.solr.client.solrj.io.stream.PushBackStream.toExplanation(PushBackStream.java:56)
> at
> org.apache.solr.client.solrj.io.stream.MergeStream.toExplanation(MergeStream.java:136)
> at
> org.apache.solr.client.solrj.io.stream.HashJoinStream.toExplanation(HashJoinStream.java:174)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExplanation(SelectStream.java:159)
> at
> org.apache.solr.client.solrj.io.stream.RankStream.toExplanation(RankStream.java:142)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExplanation(SelectStream.java:159)
> at
> org.apache.solr.handler.StreamHandler.handleRequestBody(StreamHandler.java:205)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.Con
[jira] [Commented] (SOLR-9691) Streaming expressions need to be able to use a metric computed by the facet stream as a field in other streams.
[
https://issues.apache.org/jira/browse/SOLR-9691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605650#comment-15605650
]
Dennis Gove commented on SOLR-9691:
---
Seems to throw on
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/io/stream/metrics/SumMetric.java#L69
Is it at all possible for the value count(\*) to be null? I have to suspect
that the extracted columnName within SumMetric isn't the expected "count(\*)"
but is something else.
> Streaming expressions need to be able to use a metric computed by the facet
> stream as a field in other streams.
> ---
>
> Key: SOLR-9691
> URL: https://issues.apache.org/jira/browse/SOLR-9691
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Parallel SQL
>Affects Versions: 6.2.1
>Reporter: Timothy Potter
>
> Using 6.2.1, I want to use a facet stream to do an intermediate count and
> then sum up the counts in a rollup stream, i.e. something like:
> {code}
> rollup(
> sort(
> hashJoin(
> search(products,
> q="*:*",
> fl="product_id,model_num",
> sort="product_id asc",
> qt="/export",
> partitionKeys="product_id"),
> hashed=facet(transactions, q="*:*", buckets="product_id",
> bucketSizeLimit=100, bucketSorts="product_id asc", count(*)),
> on="product_id"
> ),
> by="model_num asc"
> ),
> over="model_num",
> sum(count(*))
> )
> {code}
> Basically, I want to get a count of each product_id from the transactions
> collection (# of transactions per product) and then join that with the
> products table to generate a projection containing:
> {code}
> {
> "result-set": {
> "docs": [
> {
> "product_id": "1234",
> "count(*)": 4,
> "model_num": "X"
> },
> {
> "product_id": "5678",
> "count(*)": 5,
> "model_num": "Y"
> },
> ...
> ]
> }
> }
> {code}
> This works, but the outer rollup doesn't recognize the count(*) as a field. I
> get this error:
> {code}
> {
> "result-set": {
> "docs": [
> {
> "EXCEPTION": "Invalid expression sum(count(*)) - expected
> sum(columnName)",
> "EOF": true
> }
> ]
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9691) Streaming expressions need to be able to use a metric computed by the facet stream as a field in other streams.
[
https://issues.apache.org/jira/browse/SOLR-9691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605474#comment-15605474
]
Dennis Gove commented on SOLR-9691:
---
I suspect it's seeing count(\*) and considering it an expression representing
CountMetric. Try quoting the field name to get around this
{code}
sum("count(*)")
{code}
> Streaming expressions need to be able to use a metric computed by the facet
> stream as a field in other streams.
> ---
>
> Key: SOLR-9691
> URL: https://issues.apache.org/jira/browse/SOLR-9691
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Parallel SQL
>Affects Versions: 6.2.1
>Reporter: Timothy Potter
>
> Using 6.2.1, I want to use a facet stream to do an intermediate count and
> then sum up the counts in a rollup stream, i.e. something like:
> {code}
> rollup(
> sort(
> hashJoin(
> search(products,
> q="*:*",
> fl="product_id,model_num",
> sort="product_id asc",
> qt="/export",
> partitionKeys="product_id"),
> hashed=facet(transactions, q="*:*", buckets="product_id",
> bucketSizeLimit=100, bucketSorts="product_id asc", count(*)),
> on="product_id"
> ),
> by="model_num asc"
> ),
> over="model_num",
> sum(count(*))
> )
> {code}
> Basically, I want to get a count of each product_id from the transactions
> collection (# of transactions per product) and then join that with the
> products table to generate a projection containing:
> {code}
> {
> "result-set": {
> "docs": [
> {
> "product_id": "1234",
> "count(*)": 4,
> "model_num": "X"
> },
> {
> "product_id": "5678",
> "count(*)": 5,
> "model_num": "Y"
> },
> ...
> ]
> }
> }
> {code}
> This works, but the outer rollup doesn't recognize the count(*) as a field. I
> get this error:
> {code}
> {
> "result-set": {
> "docs": [
> {
> "EXCEPTION": "Invalid expression sum(count(*)) - expected
> sum(columnName)",
> "EOF": true
> }
> ]
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9661) Explain of select that uses replace() throws exception
[
https://issues.apache.org/jira/browse/SOLR-9661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15586661#comment-15586661
]
Dennis Gove commented on SOLR-9661:
---
I haven't looked at the code but I can imagine how this would occur. I'll see
if I can take a look in the next day or two.
> Explain of select that uses replace() throws exception
> --
>
> Key: SOLR-9661
> URL: https://issues.apache.org/jira/browse/SOLR-9661
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Gus Heck
>
> {code}
> select(
>search(test, q="table:article ", fl="edge_id", sort="edge_id desc",
> rows=10),
>edge_id,
>replace(type,null, withValue="1")
> )
> {code}
> as a streaming expression produced this stack trace:
> {code}
> ERROR (qtp1989972246-17) [c:test s:shard1 r:core_node1
> x:test_shard1_replica1] o.a.s.s.HttpSolrCall null:java.io.IOException: Unable
> to find function name for class
> 'org.apache.solr.client.solrj.io.ops.ReplaceWithValueOperation'
> at
> org.apache.solr.client.solrj.io.stream.expr.StreamFactory.getFunctionName(StreamFactory.java:335)
> at
> org.apache.solr.client.solrj.io.ops.ReplaceWithValueOperation.toExpression(ReplaceWithValueOperation.java:108)
> at
> org.apache.solr.client.solrj.io.ops.ReplaceOperation.toExpression(ReplaceOperation.java:81)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExpression(SelectStream.java:148)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExplanation(SelectStream.java:164)
> at
> org.apache.solr.client.solrj.io.stream.PushBackStream.toExplanation(PushBackStream.java:56)
> at
> org.apache.solr.client.solrj.io.stream.ComplementStream.toExplanation(ComplementStream.java:132)
> at
> org.apache.solr.client.solrj.io.stream.PushBackStream.toExplanation(PushBackStream.java:56)
> at
> org.apache.solr.client.solrj.io.stream.ComplementStream.toExplanation(ComplementStream.java:132)
> at
> org.apache.solr.client.solrj.io.stream.RankStream.toExplanation(RankStream.java:142)
> at
> org.apache.solr.client.solrj.io.stream.PushBackStream.toExplanation(PushBackStream.java:56)
> at
> org.apache.solr.client.solrj.io.stream.MergeStream.toExplanation(MergeStream.java:136)
> at
> org.apache.solr.client.solrj.io.stream.HashJoinStream.toExplanation(HashJoinStream.java:174)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExplanation(SelectStream.java:159)
> at
> org.apache.solr.client.solrj.io.stream.RankStream.toExplanation(RankStream.java:142)
> at
> org.apache.solr.client.solrj.io.stream.SelectStream.toExplanation(SelectStream.java:159)
> at
> org.apache.solr.handler.StreamHandler.handleRequestBody(StreamHandler.java:205)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:154)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2089)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.
[jira] [Commented] (SOLR-9077) Streaming expression in solr doesnot support collection alias
[
https://issues.apache.org/jira/browse/SOLR-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582274#comment-15582274
]
Dennis Gove commented on SOLR-9077:
---
It might be necessary to do that in each source stream, though a centralized
location might be better. The StreamFactory could be a place to store the
aliases after looking them up but I'm not 100% sure that's the best place.
> Streaming expression in solr doesnot support collection alias
> -
>
> Key: SOLR-9077
> URL: https://issues.apache.org/jira/browse/SOLR-9077
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 5.5.1
>Reporter: Suds
>Priority: Minor
>
> Streaming expression in solr does not support collection alias
> when I tried to access collection alias I get null pointer exception
> issue seems to be related to following code , clusterState.getActiveSlices
> returns null
> Collection slices = clusterState.getActiveSlices(this.collection);
> for(Slice slice : slices) {
> }
> fix seems to fairly simple , clusterState.getActiveSlices can be made aware
> of collection alias. I am not sure what will happen when we have large alias
> which has hundred of slices.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-9525) split() function for streaming
[
https://issues.apache.org/jira/browse/SOLR-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15577338#comment-15577338
]
Dennis Gove edited comment on SOLR-9525 at 10/15/16 4:27 AM:
-
Full implementation and tests for a split operation. Because it's implemented
as an operation this will work as part of a select() stream.
Valid expression forms:
{code}
split(fieldA, on=",") // replace value of fieldA with List of split
values
split(fieldA, on=",", as="fieldB") // splits value of fieldA into List
and puts into fieldB
{code}
was (Author: dpgove):
Full implementation and tests for a split operation. Because it's implementing
as an operation this will work as part of a select() stream.
Valid expression forms:
{code}
split(fieldA, on=",") // replace value of fieldA with List of split
values
split(fieldA, on=",", as="fieldB") // splits value of fieldA into List
and puts into fieldB
{code}
> split() function for streaming
> --
>
> Key: SOLR-9525
> URL: https://issues.apache.org/jira/browse/SOLR-9525
> Project: Solr
> Issue Type: Wish
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mike Thomsen
> Attachments: SOLR-9525.patch
>
>
> This is the original description I posted on solr-user:
> Read this article and thought it could be interesting as a way to do
> ingestion:
> https://dzone.com/articles/solr-streaming-expressions-for-collection-auto-upd-1
> Example from the article:
> daemon(id="12345",
> runInterval="6",
> update(users,
> batchSize=10,
> jdbc(connection="jdbc:mysql://localhost/users?user=root&password=solr",
> sql="SELECT id, name FROM users", sort="id asc",
> driver="com.mysql.jdbc.Driver")
> )
> What's the best way to handle a multivalue field using this API? Is there a
> way to tokenize something returned in a database field?
> Joel Bernstein responded with this:
> Unfortunately there currently isn't a way to split a field. But this would
> be nice functionality to add.
> The approach would be to an add a split operation that would be used by the
> select() function. It would look like this:
> select(jdbc(...), split(fieldA, delim=","), ...)
> This would make a good jira issue.
> So the TL;DR version is that I need the ability to specify in such a
> streaming operation certain fields to tokenize into multivalue fields. In one
> schema I may have to support, there are probably a half a dozen such fields.
> Perhaps I am missing a feature here, but until this is done it looks like
> this new capability cannot handle multivalue fields until something like this
> is in place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9525) split() function for streaming
[
https://issues.apache.org/jira/browse/SOLR-9525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-9525:
--
Attachment: SOLR-9525.patch
Full implementation and tests for a split operation. Because it's implementing
as an operation this will work as part of a select() stream.
Valid expression forms:
{code}
split(fieldA, on=",") // replace value of fieldA with List of split
values
split(fieldA, on=",", as="fieldB") // splits value of fieldA into List
and puts into fieldB
{code}
> split() function for streaming
> --
>
> Key: SOLR-9525
> URL: https://issues.apache.org/jira/browse/SOLR-9525
> Project: Solr
> Issue Type: Wish
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mike Thomsen
> Attachments: SOLR-9525.patch
>
>
> This is the original description I posted on solr-user:
> Read this article and thought it could be interesting as a way to do
> ingestion:
> https://dzone.com/articles/solr-streaming-expressions-for-collection-auto-upd-1
> Example from the article:
> daemon(id="12345",
> runInterval="6",
> update(users,
> batchSize=10,
> jdbc(connection="jdbc:mysql://localhost/users?user=root&password=solr",
> sql="SELECT id, name FROM users", sort="id asc",
> driver="com.mysql.jdbc.Driver")
> )
> What's the best way to handle a multivalue field using this API? Is there a
> way to tokenize something returned in a database field?
> Joel Bernstein responded with this:
> Unfortunately there currently isn't a way to split a field. But this would
> be nice functionality to add.
> The approach would be to an add a split operation that would be used by the
> select() function. It would look like this:
> select(jdbc(...), split(fieldA, delim=","), ...)
> This would make a good jira issue.
> So the TL;DR version is that I need the ability to specify in such a
> streaming operation certain fields to tokenize into multivalue fields. In one
> schema I may have to support, there are probably a half a dozen such fields.
> Perhaps I am missing a feature here, but until this is done it looks like
> this new capability cannot handle multivalue fields until something like this
> is in place.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9103) Restore ability for users to add custom Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567106#comment-15567106
]
Dennis Gove commented on SOLR-9103:
---
Cao, thank you very much for this. I'm really happy to see this feature
restored.
> Restore ability for users to add custom Streaming Expressions
> -
>
> Key: SOLR-9103
> URL: https://issues.apache.org/jira/browse/SOLR-9103
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Dennis Gove
> Fix For: 6.3
>
> Attachments: HelloStream.class, SOLR-9103.PATCH, SOLR-9103.PATCH,
> SOLR-9103.patch, SOLR-9103.patch
>
>
> StreamHandler is an implicit handler. So to make it extensible, we can
> introduce the below syntax in solrconfig.xml.
> {code}
>
> {code}
> This will add hello function to streamFactory of StreamHandler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-9103) Restore ability for users to add custom Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove resolved SOLR-9103.
---
Resolution: Fixed
Fix Version/s: 6.3
> Restore ability for users to add custom Streaming Expressions
> -
>
> Key: SOLR-9103
> URL: https://issues.apache.org/jira/browse/SOLR-9103
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
> Assignee: Dennis Gove
> Fix For: 6.3
>
> Attachments: HelloStream.class, SOLR-9103.PATCH, SOLR-9103.PATCH,
> SOLR-9103.patch, SOLR-9103.patch
>
>
> StreamHandler is an implicit handler. So to make it extensible, we can
> introduce the below syntax in solrconfig.xml.
> {code}
>
> {code}
> This will add hello function to streamFactory of StreamHandler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9103) Restore ability for users to add custom Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-9103:
--
Attachment: SOLR-9103.patch
Updated with some pre-commit changes.
> Restore ability for users to add custom Streaming Expressions
> -
>
> Key: SOLR-9103
> URL: https://issues.apache.org/jira/browse/SOLR-9103
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
> Assignee: Dennis Gove
> Attachments: HelloStream.class, SOLR-9103.PATCH, SOLR-9103.PATCH,
> SOLR-9103.patch, SOLR-9103.patch
>
>
> StreamHandler is an implicit handler. So to make it extensible, we can
> introduce the below syntax in solrconfig.xml.
> {code}
>
> {code}
> This will add hello function to streamFactory of StreamHandler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-9103) Restore ability for users to add custom Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove reassigned SOLR-9103:
-
Assignee: Dennis Gove
> Restore ability for users to add custom Streaming Expressions
> -
>
> Key: SOLR-9103
> URL: https://issues.apache.org/jira/browse/SOLR-9103
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
> Assignee: Dennis Gove
> Attachments: HelloStream.class, SOLR-9103.PATCH, SOLR-9103.PATCH,
> SOLR-9103.patch
>
>
> StreamHandler is an implicit handler. So to make it extensible, we can
> introduce the below syntax in solrconfig.xml.
> {code}
>
> {code}
> This will add hello function to streamFactory of StreamHandler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9103) Restore ability for users to add custom Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-9103:
--
Attachment: SOLR-9103.patch
Was able to get the tests passing by moving the package of HelloStream to
something that is compiled during test runs. This will prevent us from having
to include a .class file in the source code.
> Restore ability for users to add custom Streaming Expressions
> -
>
> Key: SOLR-9103
> URL: https://issues.apache.org/jira/browse/SOLR-9103
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Joel Bernstein
> Attachments: HelloStream.class, SOLR-9103.PATCH, SOLR-9103.PATCH,
> SOLR-9103.patch
>
>
> StreamHandler is an implicit handler. So to make it extensible, we can
> introduce the below syntax in solrconfig.xml.
> {code}
>
> {code}
> This will add hello function to streamFactory of StreamHandler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9103) Restore ability for users to add custom Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15562874#comment-15562874
]
Dennis Gove commented on SOLR-9103:
---
testDynamicLoadingCustomStream is not passing because it cannot find
runtimecode/HelloStream.class. Note that I did add the file
solr/core/src/test-files/runtimecode/HelloStream.java. But it doesn't appear
the test can find the .class of that. I know you provided a .class but I'm not
sure I'm comfortable adding a .class to the source code.
The test does pass if I run it directly in Eclipse, however.
{code}
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestCustomStream
-Dtests.method=testDynamicLoadingCustomStream -Dtests.seed=96673E541CBCF992
-Dtests.slow=true -Dtests.locale=fr-CH -Dtests.timezone=Europe/Sarajevo
-Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 28.5s | TestCustomStream.testDynamicLoadingCustomStream <<<
[junit4]> Throwable #1: java.lang.RuntimeException: Cannot find resource
in classpath or in file-system (relative to CWD): runtimecode/HelloStream.class
[junit4]>at
__randomizedtesting.SeedInfo.seed([96673E541CBCF992:E6D23776E646B225]:0)
[junit4]>at
org.apache.solr.SolrTestCaseJ4.getFile(SolrTestCaseJ4.java:1798)
[junit4]>at
org.apache.solr.core.TestDynamicLoading.getFileContent(TestDynamicLoading.java:261)
[junit4]>at
org.apache.solr.core.TestCustomStream.testDynamicLoadingCustomStream(TestCustomStream.java:73)
[junit4]>at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
[junit4]>at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
[junit4]>at java.lang.Thread.run(Thread.java:745)
{code}
> Restore ability for users to add custom Streaming Expressions
> -
>
> Key: SOLR-9103
> URL: https://issues.apache.org/jira/browse/SOLR-9103
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Joel Bernstein
> Attachments: HelloStream.class, SOLR-9103.PATCH, SOLR-9103.PATCH
>
>
> StreamHandler is an implicit handler. So to make it extensible, we can
> introduce the below syntax in solrconfig.xml.
> {code}
>
> {code}
> This will add hello function to streamFactory of StreamHandler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Reopened] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove reopened SOLR-8487:
---
Should've been resolved, not closed.
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.3
>Reporter: Jason Gerlowski
>Assignee: Dennis Gove
>Priority: Minor
> Fix For: 6.3
>
> Attachments: SOLR-8487.patch, SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove resolved SOLR-8487.
---
Resolution: Implemented
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.3
>Reporter: Jason Gerlowski
> Assignee: Dennis Gove
>Priority: Minor
> Fix For: 6.3
>
> Attachments: SOLR-8487.patch, SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9337) Add fetch Streaming Expression
[
https://issues.apache.org/jira/browse/SOLR-9337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513889#comment-15513889
]
Dennis Gove commented on SOLR-9337:
---
How does a fetch differ from an innerJoin? I guess it could if it read in a
tuple from the source and then looked up its specific fields, but I dunno how
performant that'd be.
> Add fetch Streaming Expression
> --
>
> Key: SOLR-9337
> URL: https://issues.apache.org/jira/browse/SOLR-9337
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Joel Bernstein
>
> The fetch() Streaming Expression wraps another expression and fetches
> additional fields for documents in batches. The fetch() expression will
> stream out the Tuples after the data has been fetched. Fields can be fetched
> from any SolrCloud collection.
> Sample syntax:
> {code}
> daemon(
>update(collectionC, batchSize="100"
> fetch(collectionB,
> topic(checkpoints, collectionA, q="*:*", fl="a,b,c",
> rows="50"),
> fl="j,m,z",
> on="a=j")))
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15513872#comment-15513872
]
Dennis Gove commented on SOLR-8487:
---
Added a section in the reference guide -
https://cwiki.apache.org/confluence/display/solr/Streaming+Expressions#StreamingExpressions-commit
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.3
>Reporter: Jason Gerlowski
>Assignee: Dennis Gove
>Priority: Minor
> Fix For: 6.3
>
> Attachments: SOLR-8487.patch, SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove closed SOLR-8487.
-
Resolution: Fixed
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.3
>Reporter: Jason Gerlowski
> Assignee: Dennis Gove
>Priority: Minor
> Fix For: 6.3
>
> Attachments: SOLR-8487.patch, SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-8487:
--
Affects Version/s: (was: 6.0)
6.3
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.3
>Reporter: Jason Gerlowski
> Assignee: Dennis Gove
>Priority: Minor
> Fix For: 6.3
>
> Attachments: SOLR-8487.patch, SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-8487:
--
Fix Version/s: (was: 6.0)
6.3
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.3
>Reporter: Jason Gerlowski
> Assignee: Dennis Gove
>Priority: Minor
> Fix For: 6.3
>
> Attachments: SOLR-8487.patch, SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove reassigned SOLR-8487:
-
Assignee: Dennis Gove
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.0
>Reporter: Jason Gerlowski
> Assignee: Dennis Gove
>Priority: Minor
> Fix For: 6.0
>
> Attachments: SOLR-8487.patch, SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-8487:
--
Attachment: SOLR-8487.patch
Properly handles the UpdateStream summary tuple and takes that into account
when sizing batches. The CommitStream will not swallow any tuples (ie, all
tuples it reads, whether summary or not, will be returned to any wrapping
stream).
Adds commit, parallelCommit, and daemonParallelCommit tests.
I believe this is ready to go.
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.0
>Reporter: Jason Gerlowski
>Priority: Minor
> Fix For: 6.0
>
> Attachments: SOLR-8487.patch, SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487515#comment-15487515
]
Dennis Gove commented on SOLR-8487:
---
I just realized I had a fundamental misunderstanding of the UpdateStream. I
thought it was returning all source tuples on a call to read() but that is not
the case. It is instead sending a batch of source tuples into the destination
collection, dropping them, and then returning a summary tuple.
This will change some of the implementation details of the CommitStream.
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.0
>Reporter: Jason Gerlowski
>Priority: Minor
> Fix For: 6.0
>
> Attachments: SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15439256#comment-15439256
]
Dennis Gove commented on SOLR-8487:
---
I'm not a huge fan of tying two streams together like that (ie, one is
dependent on the other). If we wanted to tie update and commit more closely I'd
rather see the commit as an operation inside the UpdateStream like
{code}
update(foo, stream(...), batchSize=#, commit(nBatches/batchSize/time))
{code}
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.0
>Reporter: Jason Gerlowski
>Priority: Minor
> Fix For: 6.0
>
> Attachments: SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-8487:
--
Attachment: SOLR-8487.patch
No tests (will add soon).
Doesn't add any summary tuples - not sure if these are necessary.
Supports setting batchSize, waitFlush, waitSearcher, and softCommit settings.
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.0
>Reporter: Jason Gerlowski
>Priority: Minor
> Fix For: 6.0
>
> Attachments: SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435610#comment-15435610
]
Dennis Gove commented on SOLR-8487:
---
I'm working on this. Hoping to have a first draft in a day or two.
> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 6.0
>Reporter: Jason Gerlowski
>Priority: Minor
> Fix For: 6.0
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.
> However, there's no way currently using the Streaming API to force a commit
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which
> collection to commit on. (Alternatively, it could recurse through
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple.
> (Alternatively, it could commit every X tuples, based on a parameter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Closed] (SOLR-9172) Refactor/rename some methods in stream.expr.StreamFactory
[ https://issues.apache.org/jira/browse/SOLR-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove closed SOLR-9172. - Resolution: Won't Fix I'm killing this. It led me down a rabbit hole that just kept going deeper and deeper and the benefits just don't outweigh the cost of completing it. > Refactor/rename some methods in stream.expr.StreamFactory > - > > Key: SOLR-9172 > URL: https://issues.apache.org/jira/browse/SOLR-9172 > Project: Solr > Issue Type: Improvement > Components: SolrJ >Reporter: Dennis Gove > Attachments: SOLR-9172.patch, SOLR-9172.patch > > > Refactors a bunch of a methods in StreamFactory to make the clearer and > easier to use. Also adds documentation for public methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9246) Errors for Streaming Expressions using JDBC (Oracle) stream source
[
https://issues.apache.org/jira/browse/SOLR-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-9246:
--
Fix Version/s: master (7.0)
6.2
6.1.1
6.0.2
> Errors for Streaming Expressions using JDBC (Oracle) stream source
> --
>
> Key: SOLR-9246
> URL: https://issues.apache.org/jira/browse/SOLR-9246
> Project: Solr
> Issue Type: Bug
>Affects Versions: 6.0.1
> Environment: Windows 7
>Reporter: Hui Liu
>Assignee: Dennis Gove
> Fix For: 6.0.2, 6.1.1, 6.2, master (7.0)
>
> Attachments: Re Errors for Streaming Expressions using JDBC (Oracle)
> stream source.txt, SOLR-9246.patch
>
>
> I have Solr 6.0.0 installed on my PC (windows 7), I was experimenting with
> ‘Streaming Expression’ by using Oracle jdbc as the
> stream source, but got 'null pointer' errors, below is the details on how to
> reproduce this error:
> 1. create a collection 'document6' which only contain long and string data
> type,
> schema.xml for Solr collection 'document6': (newly created empty collections
> with 2 shards)
> ===
>
>
>
>
> docValues="true" />
> precisionStep="0" positionIncrementGap="0"/>
>
>
>
>
>
>omitNorms="true"/>
>
>
> multiValued="false"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
>
> document_id
> document_id
>
> 2. create a new Oracle (version 11.2.0.3) table 'document6' that only contain
> columns whose jdbc type is long and string,
> create table document6
> (document_id number(12) not null,
> sender_msg_dest varchar2(256),
> recip_msg_dest varchar2(256),
> document_type varchar2(20),
> document_keyvarchar2(100));
> loaded 9 records;
> Oracle table 'document6': (newly created Oracle table with 9 records)
> =
> QA_DOCREP@qlgdb1 > desc document6
> Name Null?Type
> -
>
> DOCUMENT_ID NOT NULL NUMBER(12)
> SENDER_MSG_DESTVARCHAR2(256)
> RECIP_MSG_DEST VARCHAR2(256)
> DOCUMENT_TYPE VARCHAR2(20)
> DOCUMENT_KEY VARCHAR2(100)
> 3. tried this jdbc streaming expression in my browser, getting the error
> stack (see below)
> http://localhost:8988/solr/document6/stream?expr=jdbc(connection="jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql="SELECT
> document_id,sender_msg_dest,recip_msg_dest,document_type,document_key FROM
> document6",sort="document_id asc",driver="oracle.jdbc.driver.OracleDriver")
> errors in solr.log
> ==
> 2016-06-23 14:07:02.833 INFO (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.S.Request
> [document6_shard2_replica1] webapp=/solr path=/stream
> params={expr=jdbc(connection%3D"jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql%3D"SELECT+document_id,sender_msg_dest,recip_msg_dest,document_type,document_key+FROM+document6",sort%3D"document_id+asc",driver%3D"oracle.jdbc.driver.OracleDriver")}
> status=0 QTime=1
> 2016-06-23 14:07:05.282 ERROR (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.s.i.s.ExceptionStream
> java.lang.NullPointerException
> at
> org.apache.solr.client.solrj.io.stream.JDBCStream.read(JDBCStream.java:305)
> at
> org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:64)
> at
> org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:374)
> at
> org.apache.solr.response.TextResponseWriter.writeTupleStream(TextResponseWriter.java:305)
> at
> org.apache.solr.response.TextResponseWriter.wri
[jira] [Closed] (SOLR-9246) Errors for Streaming Expressions using JDBC (Oracle) stream source
[
https://issues.apache.org/jira/browse/SOLR-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove closed SOLR-9246.
-
Resolution: Fixed
> Errors for Streaming Expressions using JDBC (Oracle) stream source
> --
>
> Key: SOLR-9246
> URL: https://issues.apache.org/jira/browse/SOLR-9246
> Project: Solr
> Issue Type: Bug
>Affects Versions: 6.0.1
> Environment: Windows 7
>Reporter: Hui Liu
>Assignee: Dennis Gove
> Fix For: 6.0.2, 6.1.1, 6.2, master (7.0)
>
> Attachments: Re Errors for Streaming Expressions using JDBC (Oracle)
> stream source.txt, SOLR-9246.patch
>
>
> I have Solr 6.0.0 installed on my PC (windows 7), I was experimenting with
> ‘Streaming Expression’ by using Oracle jdbc as the
> stream source, but got 'null pointer' errors, below is the details on how to
> reproduce this error:
> 1. create a collection 'document6' which only contain long and string data
> type,
> schema.xml for Solr collection 'document6': (newly created empty collections
> with 2 shards)
> ===
>
>
>
>
> docValues="true" />
> precisionStep="0" positionIncrementGap="0"/>
>
>
>
>
>
>omitNorms="true"/>
>
>
> multiValued="false"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
>
> document_id
> document_id
>
> 2. create a new Oracle (version 11.2.0.3) table 'document6' that only contain
> columns whose jdbc type is long and string,
> create table document6
> (document_id number(12) not null,
> sender_msg_dest varchar2(256),
> recip_msg_dest varchar2(256),
> document_type varchar2(20),
> document_keyvarchar2(100));
> loaded 9 records;
> Oracle table 'document6': (newly created Oracle table with 9 records)
> =
> QA_DOCREP@qlgdb1 > desc document6
> Name Null?Type
> -
>
> DOCUMENT_ID NOT NULL NUMBER(12)
> SENDER_MSG_DESTVARCHAR2(256)
> RECIP_MSG_DEST VARCHAR2(256)
> DOCUMENT_TYPE VARCHAR2(20)
> DOCUMENT_KEY VARCHAR2(100)
> 3. tried this jdbc streaming expression in my browser, getting the error
> stack (see below)
> http://localhost:8988/solr/document6/stream?expr=jdbc(connection="jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql="SELECT
> document_id,sender_msg_dest,recip_msg_dest,document_type,document_key FROM
> document6",sort="document_id asc",driver="oracle.jdbc.driver.OracleDriver")
> errors in solr.log
> ==
> 2016-06-23 14:07:02.833 INFO (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.S.Request
> [document6_shard2_replica1] webapp=/solr path=/stream
> params={expr=jdbc(connection%3D"jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql%3D"SELECT+document_id,sender_msg_dest,recip_msg_dest,document_type,document_key+FROM+document6",sort%3D"document_id+asc",driver%3D"oracle.jdbc.driver.OracleDriver")}
> status=0 QTime=1
> 2016-06-23 14:07:05.282 ERROR (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.s.i.s.ExceptionStream
> java.lang.NullPointerException
> at
> org.apache.solr.client.solrj.io.stream.JDBCStream.read(JDBCStream.java:305)
> at
> org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:64)
> at
> org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:374)
> at
> org.apache.solr.response.TextResponseWriter.writeTupleStream(TextResponseWriter.java:305)
> at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:167)
> at
> org.apache.solr.response.JSONWrit
[jira] [Commented] (SOLR-9246) Errors for Streaming Expressions using JDBC (Oracle) stream source
[
https://issues.apache.org/jira/browse/SOLR-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355694#comment-15355694
]
Dennis Gove commented on SOLR-9246:
---
This was also committed to the master branch. I don't know why an auto note
isn't being added here.
https://github.com/apache/lucene-solr/commit/1ae0d8d6e1394a941b65c940cb449662d7cab5d2
> Errors for Streaming Expressions using JDBC (Oracle) stream source
> --
>
> Key: SOLR-9246
> URL: https://issues.apache.org/jira/browse/SOLR-9246
> Project: Solr
> Issue Type: Bug
>Affects Versions: 6.0.1
> Environment: Windows 7
> Reporter: Hui Liu
>Assignee: Dennis Gove
> Attachments: Re Errors for Streaming Expressions using JDBC (Oracle)
> stream source.txt, SOLR-9246.patch
>
>
> I have Solr 6.0.0 installed on my PC (windows 7), I was experimenting with
> ‘Streaming Expression’ by using Oracle jdbc as the
> stream source, but got 'null pointer' errors, below is the details on how to
> reproduce this error:
> 1. create a collection 'document6' which only contain long and string data
> type,
> schema.xml for Solr collection 'document6': (newly created empty collections
> with 2 shards)
> ===
>
>
>
>
> docValues="true" />
> precisionStep="0" positionIncrementGap="0"/>
>
>
>
>
>
>omitNorms="true"/>
>
>
> multiValued="false"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
>
> document_id
> document_id
>
> 2. create a new Oracle (version 11.2.0.3) table 'document6' that only contain
> columns whose jdbc type is long and string,
> create table document6
> (document_id number(12) not null,
> sender_msg_dest varchar2(256),
> recip_msg_dest varchar2(256),
> document_type varchar2(20),
> document_keyvarchar2(100));
> loaded 9 records;
> Oracle table 'document6': (newly created Oracle table with 9 records)
> =
> QA_DOCREP@qlgdb1 > desc document6
> Name Null?Type
> -
>
> DOCUMENT_ID NOT NULL NUMBER(12)
> SENDER_MSG_DESTVARCHAR2(256)
> RECIP_MSG_DEST VARCHAR2(256)
> DOCUMENT_TYPE VARCHAR2(20)
> DOCUMENT_KEY VARCHAR2(100)
> 3. tried this jdbc streaming expression in my browser, getting the error
> stack (see below)
> http://localhost:8988/solr/document6/stream?expr=jdbc(connection="jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql="SELECT
> document_id,sender_msg_dest,recip_msg_dest,document_type,document_key FROM
> document6",sort="document_id asc",driver="oracle.jdbc.driver.OracleDriver")
> errors in solr.log
> ==
> 2016-06-23 14:07:02.833 INFO (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.S.Request
> [document6_shard2_replica1] webapp=/solr path=/stream
> params={expr=jdbc(connection%3D"jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql%3D"SELECT+document_id,sender_msg_dest,recip_msg_dest,document_type,document_key+FROM+document6",sort%3D"document_id+asc",driver%3D"oracle.jdbc.driver.OracleDriver")}
> status=0 QTime=1
> 2016-06-23 14:07:05.282 ERROR (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.s.i.s.ExceptionStream
> java.lang.NullPointerException
> at
> org.apache.solr.client.solrj.io.stream.JDBCStream.read(JDBCStream.java:305)
> at
> org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:64)
> at
> org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:374)
> at
> org.apache.solr.response.TextRespo
[jira] [Commented] (SOLR-9103) Make StreamHandler extensible
[
https://issues.apache.org/jira/browse/SOLR-9103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355598#comment-15355598
]
Dennis Gove commented on SOLR-9103:
---
This ability existed in the initial patch adding Streaming Expressions
(SOLR-7377). I guess it was removed at some point, though I wonder why. Glad to
see it added back!
> Make StreamHandler extensible
> -
>
> Key: SOLR-9103
> URL: https://issues.apache.org/jira/browse/SOLR-9103
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
> Attachments: HelloStream.class, SOLR-9103.PATCH, SOLR-9103.PATCH
>
>
> StreamHandler is an implicit handler. So to make it extensible, we can
> introduce the below syntax in solrconfig.xml.
> {code}
>
> {code}
> This will add hello function to streamFactory of StreamHandler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-9246) Errors for Streaming Expressions using JDBC (Oracle) stream source
[
https://issues.apache.org/jira/browse/SOLR-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove reassigned SOLR-9246:
-
Assignee: Dennis Gove
> Errors for Streaming Expressions using JDBC (Oracle) stream source
> --
>
> Key: SOLR-9246
> URL: https://issues.apache.org/jira/browse/SOLR-9246
> Project: Solr
> Issue Type: Bug
>Affects Versions: 6.0.1
> Environment: Windows 7
>Reporter: Hui Liu
>Assignee: Dennis Gove
> Attachments: Re Errors for Streaming Expressions using JDBC (Oracle)
> stream source.txt, SOLR-9246.patch
>
>
> I have Solr 6.0.0 installed on my PC (windows 7), I was experimenting with
> ‘Streaming Expression’ by using Oracle jdbc as the
> stream source, but got 'null pointer' errors, below is the details on how to
> reproduce this error:
> 1. create a collection 'document6' which only contain long and string data
> type,
> schema.xml for Solr collection 'document6': (newly created empty collections
> with 2 shards)
> ===
>
>
>
>
> docValues="true" />
> precisionStep="0" positionIncrementGap="0"/>
>
>
>
>
>
>omitNorms="true"/>
>
>
> multiValued="false"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
>
> document_id
> document_id
>
> 2. create a new Oracle (version 11.2.0.3) table 'document6' that only contain
> columns whose jdbc type is long and string,
> create table document6
> (document_id number(12) not null,
> sender_msg_dest varchar2(256),
> recip_msg_dest varchar2(256),
> document_type varchar2(20),
> document_keyvarchar2(100));
> loaded 9 records;
> Oracle table 'document6': (newly created Oracle table with 9 records)
> =
> QA_DOCREP@qlgdb1 > desc document6
> Name Null?Type
> -
>
> DOCUMENT_ID NOT NULL NUMBER(12)
> SENDER_MSG_DESTVARCHAR2(256)
> RECIP_MSG_DEST VARCHAR2(256)
> DOCUMENT_TYPE VARCHAR2(20)
> DOCUMENT_KEY VARCHAR2(100)
> 3. tried this jdbc streaming expression in my browser, getting the error
> stack (see below)
> http://localhost:8988/solr/document6/stream?expr=jdbc(connection="jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql="SELECT
> document_id,sender_msg_dest,recip_msg_dest,document_type,document_key FROM
> document6",sort="document_id asc",driver="oracle.jdbc.driver.OracleDriver")
> errors in solr.log
> ==
> 2016-06-23 14:07:02.833 INFO (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.S.Request
> [document6_shard2_replica1] webapp=/solr path=/stream
> params={expr=jdbc(connection%3D"jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql%3D"SELECT+document_id,sender_msg_dest,recip_msg_dest,document_type,document_key+FROM+document6",sort%3D"document_id+asc",driver%3D"oracle.jdbc.driver.OracleDriver")}
> status=0 QTime=1
> 2016-06-23 14:07:05.282 ERROR (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.s.i.s.ExceptionStream
> java.lang.NullPointerException
> at
> org.apache.solr.client.solrj.io.stream.JDBCStream.read(JDBCStream.java:305)
> at
> org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:64)
> at
> org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:374)
> at
> org.apache.solr.response.TextResponseWriter.writeTupleStream(TextResponseWriter.java:305)
> at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:167)
> at
> org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:183)
> at
> or
[jira] [Updated] (SOLR-9246) Errors for Streaming Expressions using JDBC (Oracle) stream source
[
https://issues.apache.org/jira/browse/SOLR-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dennis Gove updated SOLR-9246:
--
Attachment: SOLR-9246.patch
On an unknown java class name an exception will now be thrown with the message
{code}
Unable to determine the valueSelector for column '' (col #) of
java class '' and type ''
{code}
For example
{code}
Unable to determine the valueSelector for column 'UNSP' (col #2) of java class
'[B' and type 'BINARY'
{code}
The weird looking java class name is because there doesn't appear to be a known
java class for a BINARY type.
Due to error handling within the JDBCStream, this exception will be caught and
wrapped as the cause of an IOException. The full exception trace will look like
this
{code}
java.io.IOException: Failed to generate value selectors for sqlQuery 'select
ID,UNSP from UNSUPPORTED_COLUMNS' against JDBC connection 'jdbc:hsqldb:mem:.'
at
Caused by: java.sql.SQLException: Unable to determine the valueSelector for
column 'UNSP' (col #2) of java class '[B' and type 'BINARY'
at
{code}
> Errors for Streaming Expressions using JDBC (Oracle) stream source
> --
>
> Key: SOLR-9246
> URL: https://issues.apache.org/jira/browse/SOLR-9246
> Project: Solr
> Issue Type: Bug
>Affects Versions: 6.0.1
> Environment: Windows 7
>Reporter: Hui Liu
> Attachments: Re Errors for Streaming Expressions using JDBC (Oracle)
> stream source.txt, SOLR-9246.patch
>
>
> I have Solr 6.0.0 installed on my PC (windows 7), I was experimenting with
> ‘Streaming Expression’ by using Oracle jdbc as the
> stream source, but got 'null pointer' errors, below is the details on how to
> reproduce this error:
> 1. create a collection 'document6' which only contain long and string data
> type,
> schema.xml for Solr collection 'document6': (newly created empty collections
> with 2 shards)
> ===
>
>
>
>
> docValues="true" />
> precisionStep="0" positionIncrementGap="0"/>
>
>
>
>
>
>omitNorms="true"/>
>
>
> multiValued="false"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
>
> document_id
> document_id
>
> 2. create a new Oracle (version 11.2.0.3) table 'document6' that only contain
> columns whose jdbc type is long and string,
> create table document6
> (document_id number(12) not null,
> sender_msg_dest varchar2(256),
> recip_msg_dest varchar2(256),
> document_type varchar2(20),
> document_keyvarchar2(100));
> loaded 9 records;
> Oracle table 'document6': (newly created Oracle table with 9 records)
> =
> QA_DOCREP@qlgdb1 > desc document6
> Name Null?Type
> -
>
> DOCUMENT_ID NOT NULL NUMBER(12)
> SENDER_MSG_DESTVARCHAR2(256)
> RECIP_MSG_DEST VARCHAR2(256)
> DOCUMENT_TYPE VARCHAR2(20)
> DOCUMENT_KEY VARCHAR2(100)
> 3. tried this jdbc streaming expression in my browser, getting the error
> stack (see below)
> http://localhost:8988/solr/document6/stream?expr=jdbc(connection="jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql="SELECT
> document_id,sender_msg_dest,recip_msg_dest,document_type,document_key FROM
> document6",sort="document_id asc",driver="oracle.jdbc.driver.OracleDriver")
> errors in solr.log
> ==
> 2016-06-23 14:07:02.833 INFO (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.S.Request
> [document6_shard2_replica1] webapp=/solr path=/stream
> params={expr=jdbc(connection%3D"jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql%3D"SELECT+document_id,sender_msg_dest,recip_msg_dest,document_typ
[jira] [Commented] (SOLR-9246) Errors for Streaming Expressions using JDBC (Oracle) stream source
[
https://issues.apache.org/jira/browse/SOLR-9246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346979#comment-15346979
]
Dennis Gove commented on SOLR-9246:
---
I'm gonna go ahead and post a patch with the exception.
> Errors for Streaming Expressions using JDBC (Oracle) stream source
> --
>
> Key: SOLR-9246
> URL: https://issues.apache.org/jira/browse/SOLR-9246
> Project: Solr
> Issue Type: Bug
>Affects Versions: 6.0.1
> Environment: Windows 7
>Reporter: Hui Liu
> Attachments: Re Errors for Streaming Expressions using JDBC (Oracle)
> stream source.txt
>
>
> I have Solr 6.0.0 installed on my PC (windows 7), I was experimenting with
> ‘Streaming Expression’ by using Oracle jdbc as the
> stream source, but got 'null pointer' errors, below is the details on how to
> reproduce this error:
> 1. create a collection 'document6' which only contain long and string data
> type,
> schema.xml for Solr collection 'document6': (newly created empty collections
> with 2 shards)
> ===
>
>
>
>
> docValues="true" />
> precisionStep="0" positionIncrementGap="0"/>
>
>
>
>
>
>omitNorms="true"/>
>
>
> multiValued="false"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
> docValues="true"/>
>
> document_id
> document_id
>
> 2. create a new Oracle (version 11.2.0.3) table 'document6' that only contain
> columns whose jdbc type is long and string,
> create table document6
> (document_id number(12) not null,
> sender_msg_dest varchar2(256),
> recip_msg_dest varchar2(256),
> document_type varchar2(20),
> document_keyvarchar2(100));
> loaded 9 records;
> Oracle table 'document6': (newly created Oracle table with 9 records)
> =
> QA_DOCREP@qlgdb1 > desc document6
> Name Null?Type
> -
>
> DOCUMENT_ID NOT NULL NUMBER(12)
> SENDER_MSG_DESTVARCHAR2(256)
> RECIP_MSG_DEST VARCHAR2(256)
> DOCUMENT_TYPE VARCHAR2(20)
> DOCUMENT_KEY VARCHAR2(100)
> 3. tried this jdbc streaming expression in my browser, getting the error
> stack (see below)
> http://localhost:8988/solr/document6/stream?expr=jdbc(connection="jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql="SELECT
> document_id,sender_msg_dest,recip_msg_dest,document_type,document_key FROM
> document6",sort="document_id asc",driver="oracle.jdbc.driver.OracleDriver")
> errors in solr.log
> ==
> 2016-06-23 14:07:02.833 INFO (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.S.Request
> [document6_shard2_replica1] webapp=/solr path=/stream
> params={expr=jdbc(connection%3D"jdbc:oracle:thin:qa_docrep/abc...@lit-racq01-scan.qa.gxsonline.net:1521/qlgdb",sql%3D"SELECT+document_id,sender_msg_dest,recip_msg_dest,document_type,document_key+FROM+document6",sort%3D"document_id+asc",driver%3D"oracle.jdbc.driver.OracleDriver")}
> status=0 QTime=1
> 2016-06-23 14:07:05.282 ERROR (qtp1389647288-139) [c:document6 s:shard2
> r:core_node1 x:document6_shard2_replica1] o.a.s.c.s.i.s.ExceptionStream
> java.lang.NullPointerException
> at
> org.apache.solr.client.solrj.io.stream.JDBCStream.read(JDBCStream.java:305)
> at
> org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:64)
> at
> org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:374)
> at
> org.apache.solr.response.TextResponseWriter.writeTupleStream(TextResponseWriter.java:305)
> at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:167)
> at
> org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONRespons
[jira] [Updated] (SOLR-9172) Refactor/rename some methods in stream.expr.StreamFactory
[ https://issues.apache.org/jira/browse/SOLR-9172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Gove updated SOLR-9172: -- Attachment: SOLR-9172.patch Added documentation for all public functions. removed the need to pass a class type to functions where only one class type was ever being passed. > Refactor/rename some methods in stream.expr.StreamFactory > - > > Key: SOLR-9172 > URL: https://issues.apache.org/jira/browse/SOLR-9172 > Project: Solr > Issue Type: Improvement > Components: SolrJ > Reporter: Dennis Gove > Attachments: SOLR-9172.patch, SOLR-9172.patch > > > Refactors a bunch of a methods in StreamFactory to make the clearer and > easier to use. Also adds documentation for public methods. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
