Re: solr export get wrong results

2014-12-26 Thread Joel Bernstein
Hi Sandy, I pulled Solr 4.10.3 to see if I could recreate the issue you are seeing with export and I wasn't able to recreate the bug you are seeing. For example the following query: http://localhost:8983/solr/collection1/export?q=join_i:[50 TO 500010]&wt=json&indent=true&sort=join_i+asc&fl=jo

Re: solr export get wrong results

2014-12-26 Thread Joel Bernstein
Hi Sandy, The export handler should only return documents in JSON format. The results in your second example are in XML for format so something looks to be wrong in the configuration. Can you post what your solrconfig looks like? Joel Joel Bernstein Search Engineer at Heliosearch On Fri, Dec 26

Re: Solr performance issues

2014-12-26 Thread Otis Gospodnetic
Likely lots of disk + network IO, yes. Put SPM for Solr on your nodes to double check. Otis > On Dec 26, 2014, at 09:17, Mahmoud Almokadem wrote: > > Dears, > > We've installed a cluster of one collection of 350M documents on 3 > r3.2xlarge (60GB RAM) Amazon servers. The size of index on eac

Re: solr export get wrong results

2014-12-26 Thread Erick Erickson
I think you missed a very important part of Jack's reply: bq: I notice that you don't have distrib=false on your select, which would make your select be from all nodes, while export would only be docs from the specific node you sent the request to. And from the Reference Guide on export bq: The

Re: Loading data to FieldValueCache

2014-12-26 Thread Erick Erickson
Manohar: Please approach this cautiously. You state that you have "hundreds of states". Every 100 states will use roughly 1.2G of your filter cache. Just for this field. Plus it'll fill up the cache and they may soon be aged out anyway. Can you really afford the space? Is it really a problem that

Re: solr export get wrong results

2014-12-26 Thread Ahmet Arslan
Hi, Do you have any custom solr components deployed? May be custom response writer? Ahmet On Friday, December 26, 2014 3:26 PM, Sandy Ding wrote: Hi, Ahmet, I use libuuid for unique id and I guess there shouldn't be duplicate ids. Also, the results are not just incomplete, they are screwed

Solr performance issues

2014-12-26 Thread Mahmoud Almokadem
Dears, We've installed a cluster of one collection of 350M documents on 3 r3.2xlarge (60GB RAM) Amazon servers. The size of index on each shard is about 1.1TB and maximum storage on Amazon is 1 TB so we add 2 SSD EBS General purpose (1x1TB + 1x500GB) on each instance. Then we create logical volume

Re: solr export get wrong results

2014-12-26 Thread Sandy Ding
Hi, Ahmet, I use libuuid for unique id and I guess there shouldn't be duplicate ids. Also, the results are not just incomplete, they are screwed. 2014-12-26 20:19 GMT+08:00 Ahmet Arslan : > Hi, > > Two different things : > > If you have unique key defined document with same id override within a

Re: solr export get wrong results

2014-12-26 Thread Sandy Ding
Thanks for your reply, Jack. The export result sets are incorrect in the sense that results totally don't match the query. For example, when I query age=20(age is int type), the results contains age=14, 22... curl "http://localhost:8983/solr/pa_info/export?q=age:20&fl=id,age"; will get the follo

Re: Solr server becomes non-responsive.

2014-12-26 Thread Jack Krupansky
Either you have too little RAM on each node or too much data on each node. You may need to shard the data much more heavily so that the total work on a single query is distributed in parallel to more nodes, each node having a much smaller amount of data to work on. First, always make sure that th

Re: solr export get wrong results

2014-12-26 Thread Ahmet Arslan
Hi, Two different things : If you have unique key defined document with same id override within a single shard. Plus, uniqueIDs expected to be unique across shards. Ahmet On Friday, December 26, 2014 11:00 AM, Sandy Ding wrote: Hi, all I've recently set up a solr cluster and found that "

Re: solr export get wrong results

2014-12-26 Thread Jack Krupansky
You neglected to tell us specifically in what way the export result is incorrect. Is some of the data missing, duplicated, garbled, or... what? Provide an example and be specific about what you think is "wrong" in the results. Have you modified the default solrconfig file? I notice that you don't

Re: Loading data to FieldValueCache

2014-12-26 Thread Manohar Sripada
Thanks Toke for the explanation, I will experiment with f.state.facet.method=enum Thanks, Manohar On Fri, Dec 26, 2014 at 4:09 PM, Toke Eskildsen wrote: > Manohar Sripada [manohar...@gmail.com] wrote: > > I have 100 million documents in my index. The maxDoc here is the maximum > > Documents in

RE: Loading data to FieldValueCache

2014-12-26 Thread Toke Eskildsen
Manohar Sripada [manohar...@gmail.com] wrote: > I have 100 million documents in my index. The maxDoc here is the maximum > Documents in each shard, right? How is it determined that each entry will > occupy maxDoc/8 approximately. Assuming that it is random whether a document is part of the result

Architecture for PHP web site, Solr and an application

2014-12-26 Thread Olivier Austina
Hi, I would like to query only some fields in Solr depend on the user input as I know the fields. The user send an HTML form to the PHP website. The application get the fields and their content from the PHP web site. The application then formulate a query to Solr based on this fields and other co

Re: Loading data to FieldValueCache

2014-12-26 Thread Manohar Sripada
I have 100 million documents in my index. The maxDoc here is the maximum Documents in each shard, right? How is it determined that each entry will occupy maxDoc/8 approximately. If I have to add facet.method=enum every time in the query, how should I specify for each field separately? Like in the

solr export get wrong results

2014-12-26 Thread Sandy Ding
Hi, all I've recently set up a solr cluster and found that "export" returns different results from "select". And I confirmed that the "export" results are wrong by manually query the results. Even simple queries as follows will get different results: curl "http://localhost:8983/solr/pa_info/selec