Re: Grouping by a multivalued field

2017-05-26 Thread Rick Leir
Shacky Quote "A multivalued field is useful when there are more than one value present for the field. An easy example would be tags, there can be multiple tags that need to be indexed...". So yes, you are on the right track. Cheers -- Rick https://stackoverflow.com/questions/5800762/what-is-the-u

Re: (Tiny Index) Solr dies but not OOM

2017-05-26 Thread Shawn Heisey
On 5/26/2017 2:21 PM, Rick Leir wrote: > (off topic) Strange. Your java has a resident size much lower than its > virtual size, so I assumed it was mostly in swap. I suppose that the > virtual size could be mostly lots of readonly or unmodified pages, but > that seems unlikely. This suggests that y

Re: Solr in NAS or Network Shared Drive

2017-05-26 Thread Dave
This could be useful in a space expensive situation, although the reason I wanted to try it is multiple solr instances in one server reading one index on the ssd. This use case where on the nfs still leads to a single point of failure situation on one of the most fragile parts of a server, the d

Re: Solr in NAS or Network Shared Drive

2017-05-26 Thread Florian Gleixner
Just tested: if file metadata (last change time, access permissions ...) on NFS storage change, then all NFS clients invalidate the memory cache of the file completely. So, if your index does not get changed, caching is good on readonly slaves - the NFS client queries only file metadata sometimes.

Re: (Tiny Index) Solr dies but not OOM

2017-05-26 Thread Rick Leir
(off topic) Strange. Your java has a resident size much lower than its virtual size, so I assumed it was mostly in swap. I suppose that the virtual size could be mostly lots of readonly or unmodified pages, but that seems unlikely. This suggests that you are using some swap for java pages. But I

Re: Spread SolrCloud across two locations

2017-05-26 Thread Pushkar Raste
In my setup once DC1 comes back up make sure you start only two nodes. Now bring down the original observer and make it observer again. Bring back the third node It seems like lot of work compared to Jan's setup but you get 5 voting members instead of 3 in normal situation. On May 26, 2017 1

Re: (Tiny Index) Solr dies but not OOM

2017-05-26 Thread Robert Brown
Thanks Rick, Swap is actually turned off, but reducing the number of Perl processes is a quick win. On 26/05/17 17:06, Rick Leir wrote: Robert, Cool, perl is taking most of your memory. 12 fcgi processes, at about 8% memory each. Try changing the web server config so it just forks 2 or 4 o

RE: Terms not being indexed; not sure why

2017-05-26 Thread Chip Calhoun
Thanks everyone. I was indeed changing the type for a couple of fields in my schema, for reasons I no longer recall; a tutorial said it would resolve some kind of error, but maybe that's outdated. I can now get the results I wanted with a pretty simple boost, so I think I'm good for now. Chip

Re: Solr in NAS or Network Shared Drive

2017-05-26 Thread Robert Haschart
When the indexing solr instance finishes, it fast-copies the newly built core to a new directory on the network storage, and then does the CREATE, SWAP, UNLOAD messages. Just before starting this message, I needed to update some records and re-deploy to production, the process took less time the

Re: Solr in NAS or Network Shared Drive

2017-05-26 Thread David Hastings
so are "core" and "corebak" pointing to the same datadir or do you have the indexing solr instance keep writing to a new directory? On Fri, May 26, 2017 at 1:53 PM, Robert Haschart wrote: > The process we use to signal the read-only servers, is to submit a CREATE > request pointing to the newly

Re: Solr in NAS or Network Shared Drive

2017-05-26 Thread Robert Haschart
The process we use to signal the read-only servers, is to submit a CREATE request pointing to the newly created index, with a name like corebak, then doing a SWAP request between core and corebak, then submit an UNLOAD request for the corebak which is now pointing at the previous version. The

Impact of changing precisionStep on already indexed content

2017-05-26 Thread Jaroslaw Rozanski
Hi all, What is the impact of changing "precisionStep" without re-indexing document preceding the change? Scenario: Assume you have index with field: So `tdate` of precision 6. Now let's assume you were to change `tdate` to be of precision 8. According to [1] the change will affect how many

Re: Solr in NAS or Network Shared Drive

2017-05-26 Thread Walter Underwood
Pretty sure that master/slave was in Solr 1.2. That was very nearly ten years ago. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On May 26, 2017, at 9:52 AM, David Hastings > wrote: > > Im curious about this. when you say "and signal the three So

Changing precisionStep without re-indexing

2017-05-26 Thread Jaroslaw Rozanski
Hi all What would be an impact of having index with same field of type TrieDateField stored with precisionStep 6 and 8. Scenario: 1. Existing index where field `tdate` (TrieDateField) with precisionStep set to 6. 2. Change in field type with precisionStep 8. 3. Index new content with precisionSte

Re: (Tiny Index) Solr dies but not OOM

2017-05-26 Thread Rick Leir
Robert, Cool, perl is taking most of your memory. 12 fcgi processes, at about 8% memory each. Try changing the web server config so it just forks 2 or 4 of them. And check whether your swap device is working. With a working swap disk, maybe your system would just slow down instead of crashing. N

Re: Solr in NAS or Network Shared Drive

2017-05-26 Thread David Hastings
Im curious about this. when you say "and signal the three Solr servers when the updated index is available. " how does it send the signal? IE what command, just a reload? Also what prevents them from doing a merge on their own? Thanks On Fri, May 26, 2017 at 12:09 PM, Robert Haschart wrote:

Re: Solr in NAS or Network Shared Drive

2017-05-26 Thread Erick Erickson
Bob: I'd guess you had to fiddle with lock factories and the like, although you say that master/slave wasn't even available when you put this system together so I don't even remember what was available "way back when" ;). If it ain't broke, don't fix it applies. That said, if I were redoing the s

Re: (Tiny Index) Solr dies but not OOM

2017-05-26 Thread Robert Brown
Thanks Shawn, It's more inquisitiveness now more than anything. http://web.lavoco.com/top.png (forgot to mention mariadb on there too :) On 26/05/17 16:20, Shawn Heisey wrote: On 5/26/2017 11:01 AM, Robert Brown wrote: Let's assume I can't get more RAM - why would an index of no more than

Re: (Tiny Index) Solr dies but not OOM

2017-05-26 Thread Shawn Heisey
On 5/26/2017 11:01 AM, Robert Brown wrote: > Let's assume I can't get more RAM - why would an index of no more than > 1MB (on disk) need so much? > > (without getting into why I'm using Solr on such a small index in the > first place :) > > My docs consist of 3 text fields for searching, all other

Re: Solr in NAS or Network Shared Drive

2017-05-26 Thread Robert Haschart
We have run using this exact scenario for several years. We have three Solr servers sitting behind a load balancer, with all three accessing the same Solr index stored on read-only network addressable storage. A fourth machine is used to update the index (typically daily) and signal the thr

Re: (Tiny Index) Solr dies but not OOM

2017-05-26 Thread Robert Brown
Let's assume I can't get more RAM - why would an index of no more than 1MB (on disk) need so much? (without getting into why I'm using Solr on such a small index in the first place :) My docs consist of 3 text fields for searching, all others are strings/ints for facets and filtering, about

Re: (Tiny Index) Solr dies but not OOM

2017-05-26 Thread Erick Erickson
Or get more physical memory? Solr _likes_ memory, you won't be able to do much with only 2G physical memory.. On Fri, May 26, 2017 at 2:00 AM, Robert Brown wrote: > Thanks Rick, > > Turns out it was the kernel killing it, dmesg showed: > > Out of memory: Kill process 2647 (java) score 118 or

Re: Solr uppercase inside phrase query

2017-05-26 Thread Erick Erickson
WordDelimiterFilterFactory is probably your issue, but it _should_ be working in the exact example you gave. Did you by chance change _anything_ without re-indexing? Susheel's comment about looking at the analysis page is an excellent one. Also add &debug=query and insure that you're searching aga

Re: Data from 4.10 to 6.5.1

2017-05-26 Thread Erick Erickson
And please _really_ pay attention to Rick's comment about examining your solrconfig.xml and schema file. I strongly recommend that you do _not_ just copy them, but take the base files distributed with 6.5.1 and move any field definitions and/or changes to them rather than start with your current co

The unified highlighter html escaping. Seems rather extreme...

2017-05-26 Thread Michael Joyner
Isn't the unified html escaper a rather bit extreme in it's escaping? It makes it hard to deal with for simple post-processing. The original html escaper seems to do minimial escaping, not every non-alphabetical character it can find. Also, is there a way to control how much text is returned

Re: Spread SolrCloud across two locations

2017-05-26 Thread Susheel Kumar
Thanks, Pushkar, Make sense. Trying to understand the difference between your setup vs Jan's proposed setup. - Seems like when DC1 goes down, in your setup we have to bounce *one* from observer to non-observer while in Jan's setup *two* observers to non-observers. Anything else I am missing - W

Re: Spread SolrCloud across two locations

2017-05-26 Thread Pushkar Raste
Damn, Math is hard DC1 : 3 non observers DC2 : 2 non observers 3 + 2 = 5 non observers Observers don't participate in voting = non observers participate in voting 5 non observers = 5 votes In addition to the 2 non observer, DC2 also has an observer, which as you pointed out does not participat

Re: Data from 4.10 to 6.5.1

2017-05-26 Thread Shawn Heisey
On 5/26/2017 2:20 AM, mganeshs wrote: > I am planning the following for moving my old solr index data created in > 4.10 to new solr server with 6.5.1. Let me know whether it will work out or > not. > > * Setup Solr and Collections with version 5.5 > * Copy data folder ( in old solr server 4.10 ) to

Grouping by a multivalued field

2017-05-26 Thread shacky
Hi, I need to create a new collection on my Solr 6.1.0 cluster where every row is a "content" and every content can belong to one or many categories, which are specified in a multivalued field "categories". In my web app the user can search by categories, and if wanted it can even group results by

Re: Solr uppercase inside phrase query

2017-05-26 Thread Susheel Kumar
I mean splitOnCaseChange. On Fri, May 26, 2017 at 9:26 AM, Susheel Kumar wrote: > Make generateWordParts to 0 and see your self in the Analysis tab the > results... > > On Fri, May 26, 2017 at 3:19 AM, Chien Nguyen > wrote: > >> Hi, everyone! >> I'm getting trouble with query data from solr. Wh

Re: Solr uppercase inside phrase query

2017-05-26 Thread Susheel Kumar
Make generateWordParts to 0 and see your self in the Analysis tab the results... On Fri, May 26, 2017 at 3:19 AM, Chien Nguyen wrote: > Hi, everyone! > I'm getting trouble with query data from solr. When i query with key > "iPhone > Se" it shows results, key "iPhone se" same, but i query "iphone

Re: Spread SolrCloud across two locations

2017-05-26 Thread Susheel Kumar
>From ZK documentation, observers do not participate in vote, so Pushkar, when you said 5 nodes participate in voting, what exactly you mean? -- Observers are non-voting members of an ensemble which only hear the results of votes, not the agreement protocol that leads up to them. Per ZK document

Solr uppercase inside phrase query

2017-05-26 Thread Chien Nguyen
Hi, everyone! I'm getting trouble with query data from solr. When i query with key "iPhone Se" it shows results, key "iPhone se" same, but i query "iphone se", there is no result. I used LowerCaseFilterFactory in both index and query analyzer . What's wrong with my schema? please let me see. Thank

Re: Data from 4.10 to 6.5.1

2017-05-26 Thread Rick Leir
There have been many changes since 4.10. I would suggest that you re-index or jump through high hoops. First, compare solrconfig.xml with the new default and have a look at schema.xml. On 2017-05-26 04:20 AM, mganeshs wrote: Hi, I am planning the following for moving my old solr index data

Data from 4.10 to 6.5.1

2017-05-26 Thread mganeshs
Hi, I am planning the following for moving my old solr index data created in 4.10 to new solr server with 6.5.1. Let me know whether it will work out or not. * Setup Solr and Collections with version 5.5 * Copy data folder ( in old solr server 4.10 ) to the corresponding collection's data folder

Re: (Tiny Index) Solr dies but not OOM

2017-05-26 Thread Robert Brown
Thanks Rick, Turns out it was the kernel killing it, dmesg showed: Out of memory: Kill process 2647 (java) score 118 or sacrifice child Killed process 2647, UID 1006, (java) total-vm:2857484kB, anon-rss:227440kB, file-rss:12kB Now I just need to tell the kernel not to do that. The other thin

Re: Spread SolrCloud across two locations

2017-05-26 Thread Rick Leir
Jan, Shawn, Susheel First steps first. First, let's do a fault-tolerant cluster, then maybe a _geographically_ fault-tolerant cluster. Add another server in either DC1 or DC2, in a separate rack, with independent power etc. As Shawn says below, install the third ZK there. You would satisfy m

Re: (Tiny Index) Solr dies but not OOM

2017-05-26 Thread Rick Leir
Robert, What is at the end of solr.log when it has died? Is there anything in syslog or messages? What is the other app? Run the top command, memory screen, on Ubuntu: $ top -o RES I have never used strace(1) on Solr, but that is an option. Run Solr in strace with the appropriate options to

Re: Work-around for "indexed without position data"

2017-05-26 Thread Rick Leir
Can you reproduce this error? What are the steps you take to reproduce it? ( simple is better). cheers -- Rick On 2017-05-25 05:46 PM, Solr User wrote: This is in regards to changing a field type from string to text_en_splitting, re-indexing all documents, even optimizing to give the index a