Re: The Streaming API (Solrj.io) : id must have DocValues?

[sudsport s]

Thanks Erik for reply,

Since I was storing Id (its stored field) and after enabling docValues my
guess is it will be stored in 2 places. also as per my understanding
docValues are great when you have values which repeat. I am not sure how
beneficial it would be for uniqueId field.
I am looking at collection of few hundred billion documents , that is
reason I really want to care about expense from design phase.




On Sun, Apr 24, 2016 at 7:24 PM, Erick Erickson 
wrote:

> In a word, "yes".
>
> DocValues aren't particularly expensive, or expensive at all. The idea
> is that when you sort by a field or facet, the field has to be
> "uninverted" which builds the entire structure in Java's JVM (this is
> when the field is _not_ DocValues).
>
> DocValues essentially serialize this structure to disk. So your
> on-disk index size is larger, but that size is MMaped rather than
> stored on Java's heap.
>
> Really, the question I'd have to ask though is "why do you care about
> the expense?". If you have a functional requirement that has to be
> served by returning the id via the /export handler, you really have no
> choice.
>
> Best,
> Erick
>
>
> On Sun, Apr 24, 2016 at 9:55 AM, sudsport s  wrote:
> > I was trying to use Streaming for reading basic tuple stream. I am using
> > sort by id asc ,
> > I am getting following exception
> >
> > I am using export search handler as per
> > https://cwiki.apache.org/confluence/display/solr/Exporting+Result+Sets
> >
> > null:java.io.IOException: id must have DocValues to use this feature.
> > at
> org.apache.solr.response.SortingResponseWriter.getFieldWriters(SortingResponseWriter.java:241)
> > at
> org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:120)
> > at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
> > at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:742)
> > at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:471)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
> > at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
> > at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> > at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> > at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> > at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> > at org.eclipse.jetty.server.session.SessionHandler.doScope(
> >
> >
> > does it make sense to enable docValues for unique field? How expensive
> is it?
> >
> >
> > if I have existing collection can I update schema and optimize
> > collection to get docvalues enabled for id?
> >
> >
> > --
> >
> > Thanks
>

Re: The Streaming API (Solrj.io) : id must have DocValues?

[Erick Erickson]

In a word, "yes".

DocValues aren't particularly expensive, or expensive at all. The idea
is that when you sort by a field or facet, the field has to be
"uninverted" which builds the entire structure in Java's JVM (this is
when the field is _not_ DocValues).

DocValues essentially serialize this structure to disk. So your
on-disk index size is larger, but that size is MMaped rather than
stored on Java's heap.

Really, the question I'd have to ask though is "why do you care about
the expense?". If you have a functional requirement that has to be
served by returning the id via the /export handler, you really have no
choice.

Best,
Erick


On Sun, Apr 24, 2016 at 9:55 AM, sudsport s  wrote:
> I was trying to use Streaming for reading basic tuple stream. I am using
> sort by id asc ,
> I am getting following exception
>
> I am using export search handler as per
> https://cwiki.apache.org/confluence/display/solr/Exporting+Result+Sets
>
> null:java.io.IOException: id must have DocValues to use this feature.
> at 
> org.apache.solr.response.SortingResponseWriter.getFieldWriters(SortingResponseWriter.java:241)
> at 
> org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:120)
> at 
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
> at 
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:742)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:471)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(
>
>
> does it make sense to enable docValues for unique field? How expensive is it?
>
>
> if I have existing collection can I update schema and optimize
> collection to get docvalues enabled for id?
>
>
> --
>
> Thanks

issues doing a spatial query

[GW]

I was not getting the results I expected so I started testing with the solr
webclient

Maybe I don;t understand things.

simple test query

q=*:*=locations:[49,-92 TO 50,-93]

I don't understand why I get a result set for longitude range -92 to -93
but should be zero results as far as I understand.



{
  "responseHeader": {
"status": 0,
"QTime": 2,
"params": {
  "q": "*:*",
  "indent": "true",
  "fq": "locations:[49,-92 TO 50,-93]",
  "wt": "json",
  "_": "1461541195102"
}
  },
  "response": {
"numFound": 85,
"start": 0,
"docs": [
  {
"id": "data.spidersilk.co!337",
"entity_id": "337",
"type_id": "simple",
"gender": "Male",
"name": "Aviator Sunglasses",
"short_description": "A timeless accessory staple, the
unmistakable teardrop lenses of our Aviator sunglasses appeal to
everyone from suits to rock stars to citizens of the world.",
"description": "Gunmetal frame with crystal gradient
polycarbonate lenses in grey. ",
"size": "",
"color": "",
"zdomain": "magento.spidersilk.co",
"zurl":
"http://magento.spidersilk.co/index.php/catalog/product/view/id/337/s/aviator-sunglasses/;,
"main_image_url":
"http://magento.spidersilk.co/media/catalog/product/cache/0/image/9df78eab33525d08d6e5fb8d27136e95/a/c/ace000a_1.jpg;,
"keywords": "Eyewear  ",
"data_size": "851,564",
"category": "Eyewear",
"final_price_without_tax": "295,USD",
"image_url": [
  
"http://magento.spidersilk.co/media/catalog/product/a/c/ace000a_1.jpg;,
  "http://magento.spidersilk.co/media/catalog/product/a/c/ace000b_1.jpg;
],
"locations": [
  "37.4463603,-122.1591775",
  "42.5857514,-82.8873787",
  "41.6942622,-86.2697108",
  "49.8522263,-97.1390697"
],
"_version_": 1532418847465799700
  },



Thanks,

GW

RE: Solr5.5:DocValues/CopyField does not work with Atomic updates

[Karthik Ramachandran]

I have opened JIRA

https://issues.apache.org/jira/browse/SOLR-9034

I will upload the patch soon.

With Thanks & Regards
Karthik Ramachandran
CommVault
Direct: (732) 923-2197
 Please don't print this e-mail unless you really need to

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Friday, April 22, 2016 8:24 PM
To: solr-user 
Subject: Re: Solr5.5:DocValues/CopyField does not work with Atomic updates

I think I just added the right person, let us know if you don't have access 
and/or if you need access to the LUCENE JIRA.

Erick

On Fri, Apr 22, 2016 at 5:17 PM, Karthik Ramachandran 
 wrote:
> Eric
>   I have created a JIRA id (kramachand...@commvault.com).  Once I get 
> access I will create the JIRA and submit the patch.
>
> With Thanks & Regards
> Karthik Ramachandran
> CommVault
> Direct: (732) 923-2197
> P Please don't print this e-mail unless you really need to
>
>
>
> On 4/22/16, 8:04 PM, "Erick Erickson"  wrote:
>
>>Karthik:
>>
>>The Apache mailing list is pretty aggressive about removing 
>>attachments. Could you possibly open a JIRA and attach the file as a 
>>patch? If at all possible a patch file with just the diffs would be 
>>best.
>>
>>One problem is that it'll be a two-step process. The JIRAs have been 
>>being hit with spam, so you'll have to request access once you create 
>>a JIRA ID (this list would be fine).
>>
>>Best,
>>Erick
>>
>>On Thu, Apr 21, 2016 at 9:09 PM, Karthik Ramachandran 
>> wrote:
>>> We feel the issue is in 
>>>RealTimeGetComponent.getInputDocument(SolrCore
>>>core,
>>> BytesRef idBytes) where solr calls getNonStoredDVs and add the 
>>>fields to the  original document without excluding the copyFields.
>>>
>>>
>>>
>>> We made changes to send the filteredList to 
>>>searcher.decorateDocValueFields
>>> and it started working.
>>>
>>>
>>>
>>> Attached is the modified file.
>>>
>>>
>>>
>>> With Thanks & Regards
>>> Karthik Ramachandran
>>> CommVault
>>> P Please don't print this e-mail unless you really need to
>>>
>>>
>>>
>>> -Original Message-
>>> From: Karthik Ramachandran [mailto:mrk...@gmail.com]
>>> Sent: Friday, April 22, 2016 12:08 AM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: Solr5.5:DocValues/CopyField does not work with Atomic 
>>>updates
>>>
>>>
>>>
>>> We are trying to update Field A.
>>>
>>>
>>>
>>>
>>>
>>> -Karthik
>>>
>>>
>>>
>>> On Thu, Apr 21, 2016 at 10:36 PM, John Bickerstaff 
>>>>>
 wrote:
>>>
>>>
>>>
 Which field do you try to atomically update?  A or B or some other?
>>>
 On Apr 21, 2016 8:29 PM, "Tirthankar Chatterjee" <
>>>
 tchatter...@commvault.com>
>>>
 wrote:
>>>

>>>
 > Hi,
>>>
 > Here is the scenario for SOLR5.5:
>>>
 >
>>>
 > FieldA type= stored=true indexed=true
>>>
 >
>>>
 > FieldB type= stored=false indexed=true docValue=true
>>>
 > usedocvalueasstored=false
>>>
 >
>>>
 > FieldA copyTo FieldB
>>>
 >
>>>
 > Try an Atomic update and we are getting this error:
>>>
 >
>>>
 > possible analysis error: DocValuesField "mtmround" appears more 
 > than
>>>
 > once in this document (only one value is allowed per field)
>>>
 >
>>>
 > How do we resolve this.
>>>
 >
>>>
 >
>>>
 >
>>>
 > ***Legal
>>>
 > Disclaimer***
>>>
 > "This communication may contain confidential and privileged 
 > material
>>>
 > for the sole use of the intended recipient. Any unauthorized 
 > review,
>>>
 > use or distribution by others is strictly prohibited. If you have
>>>
 > received the message by mistake, please advise the sender by 
 > reply
>>>
 > email and delete the message. Thank
>>>
 you."
>>>
 > *
 > ***
>>>
 > **
>>>

>>>
>>> ***Legal 
>>>Disclaimer***
>>> "This communication may contain confidential and privileged material 
>>>for the  sole use of the intended recipient. Any unauthorized review, 
>>>use or  distribution  by others is strictly prohibited. If you have 
>>>received the message by  mistake,  please advise the sender by reply 
>>>email and delete the message. Thank you."
>>> 
>>>*
>>>*
>>
>
>
>
>
> ***Legal Disclaimer***
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply email and 
> delete the message. Thank you."
> **

Re: Return some fields plus count???

[Ahmet Arslan]

Hi Fabricio,

May be use faceting? facet.field=parent

Ahmet


On Sunday, April 24, 2016 9:20 PM, Fabrício Quintela  
wrote:
Hi all,

Does anybody knows how to retrieve a list of fields plus a calculated field
like a subquery?

Scenario:

docs: [
{
  id: 123
  text:"AAA"
  parent:""
},
{
  id: 456
  text:"BBB"
  parent:"123"
},
id: 789
  text:"CCC"
  parent:"123"
]

My query fl=id, text, parent, count should return 123, "AAA", "", 2 when I
search by the id=123

Thanks all

how to retrieve json facet using solrj

[Yangrui Guo]

Hello

I use json facet api to get facets. The response returned with facets and
counts However, when I called the getFacetFields method in SolrJ client, I
got null results. How can I get the facet results from solrj? I set my
query as query.setParam("json.facet", "{entities : {type: terms,field:
class2} }" Am I missing something? Thanks.

Yangrui

Return some fields plus count???

[Fabrício Quintela]

Hi all,

Does anybody knows how to retrieve a list of fields plus a calculated field
like a subquery?

Scenario:

docs: [
{
  id: 123
  text:"AAA"
  parent:""
},
{
  id: 456
  text:"BBB"
  parent:"123"
},
id: 789
  text:"CCC"
  parent:"123"
]

My query fl=id, text, parent, count should return 123, "AAA", "", 2 when I
search by the id=123

Thanks all

The Streaming API (Solrj.io) : id must have DocValues?

[sudsport s]

I was trying to use Streaming for reading basic tuple stream. I am using
sort by id asc ,
I am getting following exception

I am using export search handler as per
https://cwiki.apache.org/confluence/display/solr/Exporting+Result+Sets

null:java.io.IOException: id must have DocValues to use this feature.
at 
org.apache.solr.response.SortingResponseWriter.getFieldWriters(SortingResponseWriter.java:241)
at 
org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:120)
at 
org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
at 
org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:742)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:471)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:214)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(


does it make sense to enable docValues for unique field? How expensive is it?


if I have existing collection can I update schema and optimize
collection to get docvalues enabled for id?


--

Thanks

Mockito issues with private SolrTestCaseJ4.beforeClass

[Georg Sorst]

Hi list,

I just ran into some issues with Mockito and SolrTestCaseJ4. It looks like
this:

* Mockito requires all @BeforeClass methods in the class hierarchy to be
"public static void"
* SolrTestCaseJ4.beforeClass (which is @BeforeClass) is "private static
void"
* So I cannot use Mockito as a test runner when my tests are derived from
SolrTestCaseJ4

Is there a specific reason why it is private? Am I missing something? I'll
gladly open a JIRA issue if someone can confirm that there is no good
reason for it.

Best,
Georg
-- 
*Georg M. Sorst I CTO*
FINDOLOGIC GmbH



Jakob-Haringer-Str. 5a | 5020 Salzburg I T.: +43 662 456708
E.: g.so...@findologic.com
www.findologic.com Folgen Sie uns auf: XING
facebook
 Twitter


Wir sehen uns auf dem *Shopware Community Day in Ahaus am 20.05.2016!* Hier
 Termin
vereinbaren!
Wir sehen uns auf der* dmexco in Köln am 14.09. und 15.09.2016!* Hier
 Termin
vereinbaren!