Re: Json object values in solr string field
Thanks Alex/Shawn,
Yeah currently we handling by writing some custom code from the response
and calculating the assets, but we lossing the power of default stats and
facet features when going with this approach.
Also actually it's not duplicate data, but as per our current design the
data resides like 2 docs for one account that we are planning to compress
at the same time need to use stats and facet. I know it's quite complicated
if we need to achieve both at the same time, i thinking about it how to
solve.
On Thu, Sep 27, 2018, 11:19 AM Alexandre Rafalovitch
wrote:
> If the duplicate data is only indexed, it is not actually duplicated. It is
> only an index entry and the record ids where it shows.
>
> Regards,
> Alex
>
> On Thu, Sep 27, 2018, 10:55 AM Balanathagiri Ayyasamypalanivel, <
> bala.cit...@gmail.com> wrote:
>
> > Hi Alex, thanks, we have that set up already in place, we are thinking to
> > optimize more to resign the data to avoid these duplication.
> >
> > Regards,
> > Bala.
> >
> > On Thu, Sep 27, 2018, 10:31 AM Alexandre Rafalovitch >
> > wrote:
> >
> > > Well, my feeling is that you are going in the wrong direction. And that
> > > maybe you need to focus more on separating your - non solr - storage
> > > representation and your - solr - search oriented representation.
> > >
> > > E.g. if your issue is storage, maybe you can focus on stored=false
> > > indexed=true approach.
> > >
> > > Regards,
> > > Alex
> > >
> > > On Thu, Sep 27, 2018, 10:13 AM Balanathagiri Ayyasamypalanivel, <
> > > bala.cit...@gmail.com> wrote:
> > >
> > > > Any suggestions?
> > > > Regards,
> > > > Bala.
> > > >
> > > > On Wed, Sep 26, 2018, 2:46 PM Balanathagiri Ayyasamypalanivel <
> > > > bala.cit...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Thanks for the reply, actually we are planning to optimize the huge
> > > > volume
> > > > > of data.
> > > > >
> > > > > For example, in our current system we have as below, so we can do
> > facet
> > > > > pivot or stats to get the sum of asset_td for each acct, but the
> data
> > > > > growing lot whenever more asset getting added.
> > > > >
> > > > > Id | Accts| assetid | asset_td
> > > > > 1| Acct1 | asset1 | 20
> > > > > 2| Acct1 | asset2 | 30
> > > > > 3| Acct2 | asset3 | 10
> > > > > 4| Acct3 | asset2 | 10
> > > > >
> > > > > So we planned to change as
> > > > >
> > > > > Id | Accts | asset_s
> > > > > 1 | Acct1 | [{"asset1": "20", "asset2":"30"}]
> > > > > 2 | Acct2 | [{"asset3": "10"}]
> > > > > 3 | Acct3 | [{"asset2": "10"}]
> > > > >
> > > > > But only draw back here is we have to parse the json to do the sum
> of
> > > the
> > > > > values, is there any other way to handle this scenario.
> > > > >
> > > > > Regards,
> > > > > Bala.
> > > > >
> > > > > On Wed, Sep 26, 2018, 2:25 PM Shawn Heisey
> > > wrote:
> > > > >
> > > > >> On 9/26/2018 12:20 PM, Balanathagiri Ayyasamypalanivel wrote:
> > > > >> > Currently I am storing json object type of values in string
> field
> > in
> > > > >> solr.
> > > > >> > Using this field, in the code I am parsing json objects and
> doing
> > > sum
> > > > of
> > > > >> > the values under it.
> > > > >> >
> > > > >> > In solr, do we have any option in doing it by default when using
> > the
> > > > >> json
> > > > >> > object field values.
> > > > >>
> > > > >> Even if you have JSON-formatted strings in Solr, Solr doesn't know
> > > > >> this. It has no idea that the data is JSON, and won't be able to
> do
> > > > >> anything special with the info contained there.
> > > > >>
> > > > >> Thanks,
> > > > >> Shawn
> > > > >>
> > > > >>
> > > >
> > >
> >
>
Re: Json object values in solr string field
If the duplicate data is only indexed, it is not actually duplicated. It is
only an index entry and the record ids where it shows.
Regards,
Alex
On Thu, Sep 27, 2018, 10:55 AM Balanathagiri Ayyasamypalanivel, <
bala.cit...@gmail.com> wrote:
> Hi Alex, thanks, we have that set up already in place, we are thinking to
> optimize more to resign the data to avoid these duplication.
>
> Regards,
> Bala.
>
> On Thu, Sep 27, 2018, 10:31 AM Alexandre Rafalovitch
> wrote:
>
> > Well, my feeling is that you are going in the wrong direction. And that
> > maybe you need to focus more on separating your - non solr - storage
> > representation and your - solr - search oriented representation.
> >
> > E.g. if your issue is storage, maybe you can focus on stored=false
> > indexed=true approach.
> >
> > Regards,
> > Alex
> >
> > On Thu, Sep 27, 2018, 10:13 AM Balanathagiri Ayyasamypalanivel, <
> > bala.cit...@gmail.com> wrote:
> >
> > > Any suggestions?
> > > Regards,
> > > Bala.
> > >
> > > On Wed, Sep 26, 2018, 2:46 PM Balanathagiri Ayyasamypalanivel <
> > > bala.cit...@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Thanks for the reply, actually we are planning to optimize the huge
> > > volume
> > > > of data.
> > > >
> > > > For example, in our current system we have as below, so we can do
> facet
> > > > pivot or stats to get the sum of asset_td for each acct, but the data
> > > > growing lot whenever more asset getting added.
> > > >
> > > > Id | Accts| assetid | asset_td
> > > > 1| Acct1 | asset1 | 20
> > > > 2| Acct1 | asset2 | 30
> > > > 3| Acct2 | asset3 | 10
> > > > 4| Acct3 | asset2 | 10
> > > >
> > > > So we planned to change as
> > > >
> > > > Id | Accts | asset_s
> > > > 1 | Acct1 | [{"asset1": "20", "asset2":"30"}]
> > > > 2 | Acct2 | [{"asset3": "10"}]
> > > > 3 | Acct3 | [{"asset2": "10"}]
> > > >
> > > > But only draw back here is we have to parse the json to do the sum of
> > the
> > > > values, is there any other way to handle this scenario.
> > > >
> > > > Regards,
> > > > Bala.
> > > >
> > > > On Wed, Sep 26, 2018, 2:25 PM Shawn Heisey
> > wrote:
> > > >
> > > >> On 9/26/2018 12:20 PM, Balanathagiri Ayyasamypalanivel wrote:
> > > >> > Currently I am storing json object type of values in string field
> in
> > > >> solr.
> > > >> > Using this field, in the code I am parsing json objects and doing
> > sum
> > > of
> > > >> > the values under it.
> > > >> >
> > > >> > In solr, do we have any option in doing it by default when using
> the
> > > >> json
> > > >> > object field values.
> > > >>
> > > >> Even if you have JSON-formatted strings in Solr, Solr doesn't know
> > > >> this. It has no idea that the data is JSON, and won't be able to do
> > > >> anything special with the info contained there.
> > > >>
> > > >> Thanks,
> > > >> Shawn
> > > >>
> > > >>
> > >
> >
>
Re: Json object values in solr string field
On 9/27/2018 8:53 AM, Balanathagiri Ayyasamypalanivel wrote: Thanks Shawn for your prompt response. Actually we have to filter on the query time while calculate the score. The challenge here is we should not add the asset and put as static field in the index time. The asset needs to be calculated while query time with some filters. Solr doesn't have that ability as far as I am aware. No matter how you slice this, you'll be writing custom code to handle it. In response to another part of the thread: search engines typically involve a lot of data duplication. It's usually faster to simply duplicate data in multiple documents than to try and normalize the data like a relational database does. Thanks, Shawn
Re: Json object values in solr string field
Hi Alex, thanks, we have that set up already in place, we are thinking to
optimize more to resign the data to avoid these duplication.
Regards,
Bala.
On Thu, Sep 27, 2018, 10:31 AM Alexandre Rafalovitch
wrote:
> Well, my feeling is that you are going in the wrong direction. And that
> maybe you need to focus more on separating your - non solr - storage
> representation and your - solr - search oriented representation.
>
> E.g. if your issue is storage, maybe you can focus on stored=false
> indexed=true approach.
>
> Regards,
> Alex
>
> On Thu, Sep 27, 2018, 10:13 AM Balanathagiri Ayyasamypalanivel, <
> bala.cit...@gmail.com> wrote:
>
> > Any suggestions?
> > Regards,
> > Bala.
> >
> > On Wed, Sep 26, 2018, 2:46 PM Balanathagiri Ayyasamypalanivel <
> > bala.cit...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Thanks for the reply, actually we are planning to optimize the huge
> > volume
> > > of data.
> > >
> > > For example, in our current system we have as below, so we can do facet
> > > pivot or stats to get the sum of asset_td for each acct, but the data
> > > growing lot whenever more asset getting added.
> > >
> > > Id | Accts| assetid | asset_td
> > > 1| Acct1 | asset1 | 20
> > > 2| Acct1 | asset2 | 30
> > > 3| Acct2 | asset3 | 10
> > > 4| Acct3 | asset2 | 10
> > >
> > > So we planned to change as
> > >
> > > Id | Accts | asset_s
> > > 1 | Acct1 | [{"asset1": "20", "asset2":"30"}]
> > > 2 | Acct2 | [{"asset3": "10"}]
> > > 3 | Acct3 | [{"asset2": "10"}]
> > >
> > > But only draw back here is we have to parse the json to do the sum of
> the
> > > values, is there any other way to handle this scenario.
> > >
> > > Regards,
> > > Bala.
> > >
> > > On Wed, Sep 26, 2018, 2:25 PM Shawn Heisey
> wrote:
> > >
> > >> On 9/26/2018 12:20 PM, Balanathagiri Ayyasamypalanivel wrote:
> > >> > Currently I am storing json object type of values in string field in
> > >> solr.
> > >> > Using this field, in the code I am parsing json objects and doing
> sum
> > of
> > >> > the values under it.
> > >> >
> > >> > In solr, do we have any option in doing it by default when using the
> > >> json
> > >> > object field values.
> > >>
> > >> Even if you have JSON-formatted strings in Solr, Solr doesn't know
> > >> this. It has no idea that the data is JSON, and won't be able to do
> > >> anything special with the info contained there.
> > >>
> > >> Thanks,
> > >> Shawn
> > >>
> > >>
> >
>
Re: Json object values in solr string field
Thanks Shawn for your prompt response. Actually we have to filter on the query time while calculate the score. The challenge here is we should not add the asset and put as static field in the index time. The asset needs to be calculated while query time with some filters. Regards, Bala. On Thu, Sep 27, 2018, 10:35 AM Shawn Heisey wrote: > On 9/26/2018 12:46 PM, Balanathagiri Ayyasamypalanivel wrote: > > But only draw back here is we have to parse the json to do the sum of the > > values, is there any other way to handle this scenario. > > Solr cannot do that for you. You could put this in your indexing > software -- add up the numbers and put the result into a new field in > your Solr document, so that the information is already in the index when > you do your query. This could be done with a custom Update Processor (a > Solr plugin that you would need to write), but if you already have > custom indexing software, it's probably easier to simply change that > software than to try and write a plugin. > > Thanks, > Shawn > >
Re: Json object values in solr string field
On 9/26/2018 12:46 PM, Balanathagiri Ayyasamypalanivel wrote: But only draw back here is we have to parse the json to do the sum of the values, is there any other way to handle this scenario. Solr cannot do that for you. You could put this in your indexing software -- add up the numbers and put the result into a new field in your Solr document, so that the information is already in the index when you do your query. This could be done with a custom Update Processor (a Solr plugin that you would need to write), but if you already have custom indexing software, it's probably easier to simply change that software than to try and write a plugin. Thanks, Shawn
Re: Json object values in solr string field
Well, my feeling is that you are going in the wrong direction. And that
maybe you need to focus more on separating your - non solr - storage
representation and your - solr - search oriented representation.
E.g. if your issue is storage, maybe you can focus on stored=false
indexed=true approach.
Regards,
Alex
On Thu, Sep 27, 2018, 10:13 AM Balanathagiri Ayyasamypalanivel, <
bala.cit...@gmail.com> wrote:
> Any suggestions?
> Regards,
> Bala.
>
> On Wed, Sep 26, 2018, 2:46 PM Balanathagiri Ayyasamypalanivel <
> bala.cit...@gmail.com> wrote:
>
> > Hi,
> >
> > Thanks for the reply, actually we are planning to optimize the huge
> volume
> > of data.
> >
> > For example, in our current system we have as below, so we can do facet
> > pivot or stats to get the sum of asset_td for each acct, but the data
> > growing lot whenever more asset getting added.
> >
> > Id | Accts| assetid | asset_td
> > 1| Acct1 | asset1 | 20
> > 2| Acct1 | asset2 | 30
> > 3| Acct2 | asset3 | 10
> > 4| Acct3 | asset2 | 10
> >
> > So we planned to change as
> >
> > Id | Accts | asset_s
> > 1 | Acct1 | [{"asset1": "20", "asset2":"30"}]
> > 2 | Acct2 | [{"asset3": "10"}]
> > 3 | Acct3 | [{"asset2": "10"}]
> >
> > But only draw back here is we have to parse the json to do the sum of the
> > values, is there any other way to handle this scenario.
> >
> > Regards,
> > Bala.
> >
> > On Wed, Sep 26, 2018, 2:25 PM Shawn Heisey wrote:
> >
> >> On 9/26/2018 12:20 PM, Balanathagiri Ayyasamypalanivel wrote:
> >> > Currently I am storing json object type of values in string field in
> >> solr.
> >> > Using this field, in the code I am parsing json objects and doing sum
> of
> >> > the values under it.
> >> >
> >> > In solr, do we have any option in doing it by default when using the
> >> json
> >> > object field values.
> >>
> >> Even if you have JSON-formatted strings in Solr, Solr doesn't know
> >> this. It has no idea that the data is JSON, and won't be able to do
> >> anything special with the info contained there.
> >>
> >> Thanks,
> >> Shawn
> >>
> >>
>
Re: Json object values in solr string field
Any suggestions?
Regards,
Bala.
On Wed, Sep 26, 2018, 2:46 PM Balanathagiri Ayyasamypalanivel <
bala.cit...@gmail.com> wrote:
> Hi,
>
> Thanks for the reply, actually we are planning to optimize the huge volume
> of data.
>
> For example, in our current system we have as below, so we can do facet
> pivot or stats to get the sum of asset_td for each acct, but the data
> growing lot whenever more asset getting added.
>
> Id | Accts| assetid | asset_td
> 1| Acct1 | asset1 | 20
> 2| Acct1 | asset2 | 30
> 3| Acct2 | asset3 | 10
> 4| Acct3 | asset2 | 10
>
> So we planned to change as
>
> Id | Accts | asset_s
> 1 | Acct1 | [{"asset1": "20", "asset2":"30"}]
> 2 | Acct2 | [{"asset3": "10"}]
> 3 | Acct3 | [{"asset2": "10"}]
>
> But only draw back here is we have to parse the json to do the sum of the
> values, is there any other way to handle this scenario.
>
> Regards,
> Bala.
>
> On Wed, Sep 26, 2018, 2:25 PM Shawn Heisey wrote:
>
>> On 9/26/2018 12:20 PM, Balanathagiri Ayyasamypalanivel wrote:
>> > Currently I am storing json object type of values in string field in
>> solr.
>> > Using this field, in the code I am parsing json objects and doing sum of
>> > the values under it.
>> >
>> > In solr, do we have any option in doing it by default when using the
>> json
>> > object field values.
>>
>> Even if you have JSON-formatted strings in Solr, Solr doesn't know
>> this. It has no idea that the data is JSON, and won't be able to do
>> anything special with the info contained there.
>>
>> Thanks,
>> Shawn
>>
>>
Re: Json object values in solr string field
Hi,
Thanks for the reply, actually we are planning to optimize the huge volume
of data.
For example, in our current system we have as below, so we can do facet
pivot or stats to get the sum of asset_td for each acct, but the data
growing lot whenever more asset getting added.
Id | Accts| assetid | asset_td
1| Acct1 | asset1 | 20
2| Acct1 | asset2 | 30
3| Acct2 | asset3 | 10
4| Acct3 | asset2 | 10
So we planned to change as
Id | Accts | asset_s
1 | Acct1 | [{"asset1": "20", "asset2":"30"}]
2 | Acct2 | [{"asset3": "10"}]
3 | Acct3 | [{"asset2": "10"}]
But only draw back here is we have to parse the json to do the sum of the
values, is there any other way to handle this scenario.
Regards,
Bala.
On Wed, Sep 26, 2018, 2:25 PM Shawn Heisey wrote:
> On 9/26/2018 12:20 PM, Balanathagiri Ayyasamypalanivel wrote:
> > Currently I am storing json object type of values in string field in
> solr.
> > Using this field, in the code I am parsing json objects and doing sum of
> > the values under it.
> >
> > In solr, do we have any option in doing it by default when using the json
> > object field values.
>
> Even if you have JSON-formatted strings in Solr, Solr doesn't know
> this. It has no idea that the data is JSON, and won't be able to do
> anything special with the info contained there.
>
> Thanks,
> Shawn
>
>
Re: Json object values in solr string field
On 9/26/2018 12:20 PM, Balanathagiri Ayyasamypalanivel wrote: Currently I am storing json object type of values in string field in solr. Using this field, in the code I am parsing json objects and doing sum of the values under it. In solr, do we have any option in doing it by default when using the json object field values. Even if you have JSON-formatted strings in Solr, Solr doesn't know this. It has no idea that the data is JSON, and won't be able to do anything special with the info contained there. Thanks, Shawn
