Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()
Here it is: https://issues.apache.org/jira/browse/SOLR-12499 -- Christian Spitzlay Diplom-Physiker, Senior Software-Entwickler Tel: +49 69 / 348739116 E-Mail: christian.spitz...@biologis.com bio.logis Genetic Information Management GmbH Altenhöferallee 3 60438 Frankfurt am Main Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt Enrico Just Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945 Umsatzsteuer-Identifikationsnummer DE293587677 > Am 19.06.2018 um 15:30 schrieb Christian Spitzlay > : > > Ok. I'm about to create the issue and I have a draft version of what I had in > mind > in a branch on github. > > Christian Spitzlay > > >> Am 19.06.2018 um 15:27 schrieb Joel Bernstein : >> >> Let's move the discussion to the jira ticket. >> >> Joel Bernstein >> http://joelsolr.blogspot.com/ >> >> On Tue, Jun 19, 2018 at 3:42 AM, Christian Spitzlay < >> christian.spitz...@biologis.com> wrote: >> >>> >>> Am 18.06.2018 um 15:30 schrieb Joel Bernstein : You are doing things correctly. I was incorrect about the behavior of the group() operation. I think the behavior you are looking for should be done using reduce() >>> but we'll need to create a reduce operation that does this. If you want to create a ticket we can work through exactly how the operation would work. >>> >>> >>> I'll create an issue tonight at the latest. >>> Should we take further discussions off the user list >>> or is it acceptable here? >>> >>> >>> Christian Spitzlay >>> >>> >>> -- >>> >>> Christian Spitzlay >>> Diplom-Physiker, >>> Senior Software-Entwickler >>> >>> Tel: +49 69 / 348739116 >>> E-Mail: christian.spitz...@biologis.com >>> >>> bio.logis Genetic Information Management GmbH >>> Altenhöferallee 3 >>> 60438 Frankfurt am Main >>> >>> Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt >>> Enrico Just >>> Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945 >>> Umsatzsteuer-Identifikationsnummer DE293587677 >>> >>> >>> >>> >>> >>> >>> >
Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()
Ok. I'm about to create the issue and I have a draft version of what I had in mind in a branch on github. Christian Spitzlay > Am 19.06.2018 um 15:27 schrieb Joel Bernstein : > > Let's move the discussion to the jira ticket. > > Joel Bernstein > http://joelsolr.blogspot.com/ > > On Tue, Jun 19, 2018 at 3:42 AM, Christian Spitzlay < > christian.spitz...@biologis.com> wrote: > >> >> >>> Am 18.06.2018 um 15:30 schrieb Joel Bernstein : >>> >>> You are doing things correctly. I was incorrect about the behavior of the >>> group() operation. >>> >>> I think the behavior you are looking for should be done using reduce() >> but >>> we'll need to create a reduce operation that does this. If you want to >>> create a ticket we can work through exactly how the operation would work. >>> >> >> >> I'll create an issue tonight at the latest. >> Should we take further discussions off the user list >> or is it acceptable here? >> >> >> Christian Spitzlay >> >> >> -- >> >> Christian Spitzlay >> Diplom-Physiker, >> Senior Software-Entwickler >> >> Tel: +49 69 / 348739116 >> E-Mail: christian.spitz...@biologis.com >> >> bio.logis Genetic Information Management GmbH >> Altenhöferallee 3 >> 60438 Frankfurt am Main >> >> Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt >> Enrico Just >> Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945 >> Umsatzsteuer-Identifikationsnummer DE293587677 >> >> >> >> >> >> >>
Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()
Let's move the discussion to the jira ticket. Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Jun 19, 2018 at 3:42 AM, Christian Spitzlay < christian.spitz...@biologis.com> wrote: > > > > Am 18.06.2018 um 15:30 schrieb Joel Bernstein : > > > > You are doing things correctly. I was incorrect about the behavior of the > > group() operation. > > > > I think the behavior you are looking for should be done using reduce() > but > > we'll need to create a reduce operation that does this. If you want to > > create a ticket we can work through exactly how the operation would work. > > > > > I'll create an issue tonight at the latest. > Should we take further discussions off the user list > or is it acceptable here? > > > Christian Spitzlay > > > -- > > Christian Spitzlay > Diplom-Physiker, > Senior Software-Entwickler > > Tel: +49 69 / 348739116 > E-Mail: christian.spitz...@biologis.com > > bio.logis Genetic Information Management GmbH > Altenhöferallee 3 > 60438 Frankfurt am Main > > Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt > Enrico Just > Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945 > Umsatzsteuer-Identifikationsnummer DE293587677 > > > > > > >
Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()
> Am 18.06.2018 um 15:30 schrieb Joel Bernstein : > > You are doing things correctly. I was incorrect about the behavior of the > group() operation. > > I think the behavior you are looking for should be done using reduce() but > we'll need to create a reduce operation that does this. If you want to > create a ticket we can work through exactly how the operation would work. > I'll create an issue tonight at the latest. Should we take further discussions off the user list or is it acceptable here? Christian Spitzlay -- Christian Spitzlay Diplom-Physiker, Senior Software-Entwickler Tel: +49 69 / 348739116 E-Mail: christian.spitz...@biologis.com bio.logis Genetic Information Management GmbH Altenhöferallee 3 60438 Frankfurt am Main Geschäftsführung: Prof. Dr. med. Daniela Steinberger, Dipl.Betriebswirt Enrico Just Firmensitz Frankfurt am Main, Registergericht Frankfurt am Main, HRB 97945 Umsatzsteuer-Identifikationsnummer DE293587677
Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()
You are doing things correctly. I was incorrect about the behavior of the
group() operation.
I think the behavior you are looking for should be done using reduce() but
we'll need to create a reduce operation that does this. If you want to
create a ticket we can work through exactly how the operation would work.
Joel Bernstein
http://joelsolr.blogspot.com/
On Fri, Jun 15, 2018 at 8:04 AM, Christian Spitzlay <
christian.spitz...@biologis.com> wrote:
> Hi,
>
> I had come across the reduce function in the docs but
> I have a hard time getting it to work; I haven't found any documentation
> on it
> or its parameters, and the source code of the GroupOperation doesn't
> explain it either ...
> For example, what is the "n" parameter about?
>
>
> I constructed a source stream to produce the input from my second example:
>
> merge(
> sort(cartesianProduct(tuple(k1="1", k2=array(a)), k2, productSort="k1
> asc"), by="k1 asc"),
> sort(cartesianProduct(tuple(k1="2", k2=array(b,c)), k2,
> productSort="k1 asc"), by="k1 asc"),
> on="k1 asc"
> )
>
> --->
>
> {
> "result-set": {
> "docs": [
> {
> "k1": "1",
> "k2": "a"
> },
> {
> "k1": "2",
> "k2": "b"
> },
> {
> "k1": "2",
> "k2": "c"
> },
> {
> "EOF": true,
> "RESPONSE_TIME": 0
> }
> ]
> }
> }
>
>
>
> Then wrapped in a reduce function:
>
>
> reduce(
> merge(
> sort(cartesianProduct(tuple(k1="1", k2=array(a)), k2, productSort="k1
> asc"), by="k1 asc"),
> sort(cartesianProduct(tuple(k1="2", k2=array(b,c)), k2,
> productSort="k1 asc"), by="k1 asc"),
> on="k1 asc"
> ),
> by="k1",
> group(sort="k1 asc", n="10")
> )
>
> --->
>
> {
> "result-set": {
> "docs": [
> {
> "k1": "1",
> "k2": "a",
> "group": [
> {
> "k1": "1",
> "k2": "a"
> }
> ]
> },
> {
> "k1": "2",
> "k2": "c",
> "group": [
> {
> "k1": "2",
> "k2": "c"
> },
> {
> "k1": "2",
> "k2": "b"
> }
> ]
> },
> {
> "EOF": true,
> "RESPONSE_TIME": 0
> }
> ]
> }
> }
>
>
> It adds a field "group" that contains an array of the unchanged input
> documents with the same "by" value,
> not grouped values.
>
> {
> "result-set": {
> "docs": [
> {
> "k1": "1",
> "k2": "a",
> "group": [
> {
> "k1": "1",
> "k2": "a"
> }
> ]
> },
> {
> "k1": "2",
> "k2": "c",
> "group": [
> {
> "k1": "2",
> "k2": "c"
> },
> {
> "k1": "2",
> "k2": "b"
> }
> ]
> },
> {
> "EOF": true,
> "RESPONSE_TIME": 0
> }
> ]
> }
> }
>
>
> Or am I doing it wrong?
>
> Christian Spitzlay
>
>
>
>
>
>
>
>
>
>
> > Am 15.06.2018 um 01:48 schrieb Joel Bernstein :
> >
> > Actually you're second example is probably a straight forward:
> >
> > reduce(select(...), group(...), by="k1")
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Thu, Jun 14, 2018 at 7:33 PM, Joel Bernstein
> wrote:
> >
> >> Take a look at the reduce() function. You'll have to write a custom
> reduce
> >> operation but you can follow the example here:
> >>
> >> https://github.com/apache/lucene-solr/blob/master/solr/
> >> solrj/src/java/org/apache/solr/client/solrj/io/ops/GroupOperation.java
> >>
> >> You can plug in your custom reduce operation in the solrconfig.xml and
> use
> >> it like any other function. If you're interested in working on this you
> >> could create a ticket and I can provide guidance.
> >>
> >>
> >> Joel Bernstein
> >> http://joelsolr.blogspot.com/
> >>
> >> 2018-06-14 13:13 GMT-04:00 Christian Spitzlay <
> >> christian.spitz...@biologis.com>:
> >>
> >>> Hi,
> >>>
> >>> is there a way to merge array values?
> >>>
> >>> Something that transforms
> >>>
> >>> {
> >>> "k1": "1",
> >>> "k2": ["a", "b"]
> >>> },
> >>> {
> >>> "k1": "2",
> >>> "k2": ["c", "d"]
> >>> },
> >>> {
> >>> "k1": "2",
> >>> "k2": ["e", "f"]
> >>> }
> >>>
> >>> into
> >>>
> >>> {
> >>> "k1": "1",
> >>> "k2": ["a", "b"]
> >>> },
> >>> {
> >>> "k1": "2",
> >>> "k2": ["c", "d", "e", "f"]
> >>> }
> >>>
> >>>
> >>> And an inverse of cartesianProduct() that transforms
> >>>
> >>> {
> >>> "k1": "1",
> >>> "k2": "a"
> >>> },
> >>> {
> >>> "k1": "2",
> >>> "k2": "b"
> >>> },
> >>> {
> >>> "k1": "2",
> >>> "k2": "c"
> >>> }
> >>>
> >>> into
> >>>
> >>> {
> >>> "k1": "1",
> >>> "k2": ["a"]
> >>> },
> >>> {
> >>> "k1": "2",
> >>> "k2": ["b", "c"]
> >>> }
> >>>
> >>>
> >>> Christian
> >>>
> >>>
> >>>
> >>
>
>
Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()
Hi,
I had come across the reduce function in the docs but
I have a hard time getting it to work; I haven't found any documentation on it
or its parameters, and the source code of the GroupOperation doesn't explain it
either ...
For example, what is the "n" parameter about?
I constructed a source stream to produce the input from my second example:
merge(
sort(cartesianProduct(tuple(k1="1", k2=array(a)), k2, productSort="k1
asc"), by="k1 asc"),
sort(cartesianProduct(tuple(k1="2", k2=array(b,c)), k2, productSort="k1
asc"), by="k1 asc"),
on="k1 asc"
)
--->
{
"result-set": {
"docs": [
{
"k1": "1",
"k2": "a"
},
{
"k1": "2",
"k2": "b"
},
{
"k1": "2",
"k2": "c"
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
Then wrapped in a reduce function:
reduce(
merge(
sort(cartesianProduct(tuple(k1="1", k2=array(a)), k2, productSort="k1
asc"), by="k1 asc"),
sort(cartesianProduct(tuple(k1="2", k2=array(b,c)), k2, productSort="k1
asc"), by="k1 asc"),
on="k1 asc"
),
by="k1",
group(sort="k1 asc", n="10")
)
--->
{
"result-set": {
"docs": [
{
"k1": "1",
"k2": "a",
"group": [
{
"k1": "1",
"k2": "a"
}
]
},
{
"k1": "2",
"k2": "c",
"group": [
{
"k1": "2",
"k2": "c"
},
{
"k1": "2",
"k2": "b"
}
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
It adds a field "group" that contains an array of the unchanged input documents
with the same "by" value,
not grouped values.
{
"result-set": {
"docs": [
{
"k1": "1",
"k2": "a",
"group": [
{
"k1": "1",
"k2": "a"
}
]
},
{
"k1": "2",
"k2": "c",
"group": [
{
"k1": "2",
"k2": "c"
},
{
"k1": "2",
"k2": "b"
}
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
Or am I doing it wrong?
Christian Spitzlay
> Am 15.06.2018 um 01:48 schrieb Joel Bernstein :
>
> Actually you're second example is probably a straight forward:
>
> reduce(select(...), group(...), by="k1")
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, Jun 14, 2018 at 7:33 PM, Joel Bernstein wrote:
>
>> Take a look at the reduce() function. You'll have to write a custom reduce
>> operation but you can follow the example here:
>>
>> https://github.com/apache/lucene-solr/blob/master/solr/
>> solrj/src/java/org/apache/solr/client/solrj/io/ops/GroupOperation.java
>>
>> You can plug in your custom reduce operation in the solrconfig.xml and use
>> it like any other function. If you're interested in working on this you
>> could create a ticket and I can provide guidance.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>> 2018-06-14 13:13 GMT-04:00 Christian Spitzlay <
>> christian.spitz...@biologis.com>:
>>
>>> Hi,
>>>
>>> is there a way to merge array values?
>>>
>>> Something that transforms
>>>
>>> {
>>> "k1": "1",
>>> "k2": ["a", "b"]
>>> },
>>> {
>>> "k1": "2",
>>> "k2": ["c", "d"]
>>> },
>>> {
>>> "k1": "2",
>>> "k2": ["e", "f"]
>>> }
>>>
>>> into
>>>
>>> {
>>> "k1": "1",
>>> "k2": ["a", "b"]
>>> },
>>> {
>>> "k1": "2",
>>> "k2": ["c", "d", "e", "f"]
>>> }
>>>
>>>
>>> And an inverse of cartesianProduct() that transforms
>>>
>>> {
>>> "k1": "1",
>>> "k2": "a"
>>> },
>>> {
>>> "k1": "2",
>>> "k2": "b"
>>> },
>>> {
>>> "k1": "2",
>>> "k2": "c"
>>> }
>>>
>>> into
>>>
>>> {
>>> "k1": "1",
>>> "k2": ["a"]
>>> },
>>> {
>>> "k1": "2",
>>> "k2": ["b", "c"]
>>> }
>>>
>>>
>>> Christian
>>>
>>>
>>>
>>
Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()
Actually you're second example is probably a straight forward:
reduce(select(...), group(...), by="k1")
Joel Bernstein
http://joelsolr.blogspot.com/
On Thu, Jun 14, 2018 at 7:33 PM, Joel Bernstein wrote:
> Take a look at the reduce() function. You'll have to write a custom reduce
> operation but you can follow the example here:
>
> https://github.com/apache/lucene-solr/blob/master/solr/
> solrj/src/java/org/apache/solr/client/solrj/io/ops/GroupOperation.java
>
> You can plug in your custom reduce operation in the solrconfig.xml and use
> it like any other function. If you're interested in working on this you
> could create a ticket and I can provide guidance.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> 2018-06-14 13:13 GMT-04:00 Christian Spitzlay <
> christian.spitz...@biologis.com>:
>
>> Hi,
>>
>> is there a way to merge array values?
>>
>> Something that transforms
>>
>> {
>> "k1": "1",
>> "k2": ["a", "b"]
>> },
>> {
>> "k1": "2",
>> "k2": ["c", "d"]
>> },
>> {
>> "k1": "2",
>> "k2": ["e", "f"]
>> }
>>
>> into
>>
>> {
>> "k1": "1",
>> "k2": ["a", "b"]
>> },
>> {
>> "k1": "2",
>> "k2": ["c", "d", "e", "f"]
>> }
>>
>>
>> And an inverse of cartesianProduct() that transforms
>>
>> {
>> "k1": "1",
>> "k2": "a"
>> },
>> {
>> "k1": "2",
>> "k2": "b"
>> },
>> {
>> "k1": "2",
>> "k2": "c"
>> }
>>
>> into
>>
>> {
>> "k1": "1",
>> "k2": ["a"]
>> },
>> {
>> "k1": "2",
>> "k2": ["b", "c"]
>> }
>>
>>
>> Christian
>>
>>
>>
>
Re: Streaming Expressions: Merge array values? Inverse of cartesianProduct()
Take a look at the reduce() function. You'll have to write a custom reduce
operation but you can follow the example here:
https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/io/ops/GroupOperation.java
You can plug in your custom reduce operation in the solrconfig.xml and use
it like any other function. If you're interested in working on this you
could create a ticket and I can provide guidance.
Joel Bernstein
http://joelsolr.blogspot.com/
2018-06-14 13:13 GMT-04:00 Christian Spitzlay <
christian.spitz...@biologis.com>:
> Hi,
>
> is there a way to merge array values?
>
> Something that transforms
>
> {
> "k1": "1",
> "k2": ["a", "b"]
> },
> {
> "k1": "2",
> "k2": ["c", "d"]
> },
> {
> "k1": "2",
> "k2": ["e", "f"]
> }
>
> into
>
> {
> "k1": "1",
> "k2": ["a", "b"]
> },
> {
> "k1": "2",
> "k2": ["c", "d", "e", "f"]
> }
>
>
> And an inverse of cartesianProduct() that transforms
>
> {
> "k1": "1",
> "k2": "a"
> },
> {
> "k1": "2",
> "k2": "b"
> },
> {
> "k1": "2",
> "k2": "c"
> }
>
> into
>
> {
> "k1": "1",
> "k2": ["a"]
> },
> {
> "k1": "2",
> "k2": ["b", "c"]
> }
>
>
> Christian
>
>
>
Streaming Expressions: Merge array values? Inverse of cartesianProduct()
Hi,
is there a way to merge array values?
Something that transforms
{
"k1": "1",
"k2": ["a", "b"]
},
{
"k1": "2",
"k2": ["c", "d"]
},
{
"k1": "2",
"k2": ["e", "f"]
}
into
{
"k1": "1",
"k2": ["a", "b"]
},
{
"k1": "2",
"k2": ["c", "d", "e", "f"]
}
And an inverse of cartesianProduct() that transforms
{
"k1": "1",
"k2": "a"
},
{
"k1": "2",
"k2": "b"
},
{
"k1": "2",
"k2": "c"
}
into
{
"k1": "1",
"k2": ["a"]
},
{
"k1": "2",
"k2": ["b", "c"]
}
Christian
