Re: Replication lag after cache optimizations
Thanks for your answer Erick.
For the polling interval, we use 1 second for the small index and 1 minute
for the big one. I'll try to increase it up to 5 minutes and see if it
solve the problem.The issue doesn't occur with the default cache settings
(i.e. cache size=512).
Indeed, we strive for near real time searching, so I'll take a look at 4.0
with NRT support, hoping it is sufficiently stable for production
environment.
Kind regards,
Damien
2012/9/2 Erick Erickson erickerick...@gmail.com
You polling interval is much too short. 1 second
is probably getting you into resource contention
issues.
A more reasonable interval is on the order of several
minutes. If you really need near real time
searching, consider 4.0 which supports NRT
Best
Erick
On Fri, Aug 31, 2012 at 10:02 AM, Damien Dudognon
damien.dudog...@ebuzzing.com wrote:
Hi,
We get some troubles with the solr replication after cache
optimizations. We use a lot the facet features.
We have increased the cache size and its initial size. We have also
change the queryResultCache from LRU to FastLRU and the fieldValueCache was
activated (see below the detailed configuration). This optimization step
allows to divide the average time per request by 50 (from 260ms to 5ms).
However, with these modifications we noticed an important replication
lag. This issue is not troublesome for smaller indexes (about 300.000
elements - 3Gb), but it becomes critical if the index size is significant
(30 million elements - 70Gb). In fact, the slaves can't make up for lag and
they become out-of-date (and consequently unusable).
Has anyone ever been faced this kind of problems?
Our environment :
- Solr 3.4.0
- Java 1.6.0_26
- Debian 6.0.3
Best regards,
Damien
--
My previous cache settings (fieldValueCache was disabled):
--
filterCache class=solr.FastLRUCache size=512 initialSize=512
autowarmCount=0 /
queryResultCache class=solr.LRUCache size=512 initialSize=512
autowarmCount=0 /
documentCache class=solr.FastLRUCache size=512 initialSize=512
autowarmCount=0 /
!--
fieldValueCache class=solr.FastLRUCache size=512
autowarmCount=128 showItems=32 /
--
--
The settings now used:
--
filterCache class=solr.FastLRUCache size=16384 initialSize=4096
autowarmCount=0 /
queryResultCache class=solr.FastLRUCache size=16384
initialSize=4096 autowarmCount=0 /
documentCache class=solr.FastLRUCache size=16384 initialSize=4096
autowarmCount=0 /
fieldValueCache class=solr.FastLRUCache size=16384
autowarmCount=1024 showItems=32 /
--
The replication config:
--
requestHandler name=/replication class=solr.ReplicationHandler
lst name=master
str name=enable${solr.enable.master:false}/str
str name=replicateAftercommit/str
str name=replicateAfterstartup/str
str name=confFilesschema.xml,stopwords.txt/str
/lst
lst name=slave
str name=enable${solr.enable.slave:false}/str
str name=masterUrlhttp://solrmaster:
${jetty.port:8083}/solr/en/replication/str
str name=pollInterval00:00:01/str
/lst
/requestHandler
--
Damien DUDOGNON
RD Engineer - PhD Student
- +33 (0)6 62 79 34 10
- damien.dudog...@ebuzzing.com
-
-
- linkedin.com/in/damiendudognonhttp://www.linkedin.com/in/damiendudognon
- www.irit.fr/~Damien.Dudognon
- 1, avenue Jean Rieux
- 31500 Toulouse
- France
- +33 (0)5 62 48 33 90
-
RE: solrj api for partial document update
In the solrj api, the value of a SolrInputField can be a map, in which case solrj adds an additional attribute to the field's xml element. For example, This code: SolrInputDocument doc = new SolrInputDocument(); MapString, String partialUpdate = new HashMapString, String(); partialUpdate.put(set, foo); doc.addField(id, test_123); doc.addField(description, partialUpdate); yields this document: doc boost=1.0 field name=idtest_123/field field name=description update=setfoo/field /doc In this example I used the value set for this additional attribute, but it doesn't work. Solr doesn't update the field as I expected. According to this link: http://solr.pl/en/2012/07/09/solr-4-0-partial-documents-update/ valid values are set and add. Any idea? Thanks, Yoni -Original Message- From: Yoni Amir [mailto:yoni.a...@actimize.com] Sent: Saturday, September 01, 2012 1:48 PM To: solr-user@lucene.apache.org Subject: RE: solrj api for partial document update Any word on this? I inspected the solrj code an found nothing. It's a shame if the GA version comes out without such an api. Thanks again, Yoni -Original Message- From: Yoni Amir [mailto:yoni.a...@actimize.com] Sent: Thursday, August 30, 2012 8:48 AM To: solr-user@lucene.apache.org Subject: solrj api for partial document update Is there a solrj api for partial document update in solr 4? It is described here: http://solr.pl/en/2012/07/09/solr-4-0-partial-documents-update/ That article explains how the xml structure should be. I want to use solrj api, but I can't figure out if it is supported. Thanks, Yoni
stem porter with tokenizer..
PorterStemmer using tokenizer which class in lucene?? -- View this message in context: http://lucene.472066.n3.nabble.com/stem-porter-with-tokenizer-tp4004913.html Sent from the Solr - User mailing list archive at Nabble.com.
Solr Not releasing memory
Hi, We are running solr3.5 using tomcal 6.26 on a Windows Enterprise RC2 server, our index size if pretty large. We have noticed that once tomcat starts using/reserving ram it never releases them, even when there is not a single user on the system. I have tried forced garbage collection, but that doesn't seem to help either. Regards, Rohit
Re: LineEntityProcessor process only one file
Ahmet, please post your dih script in a message (not as an attachment). - Original Message - | From: James Dyer james.d...@ingramcontent.com | To: solr-user@lucene.apache.org | Sent: Friday, August 31, 2012 12:53:50 PM | Subject: RE: LineEntityProcessor process only one file | | No, it should process all of the files that get listed. I'm taking a | look at the issue you opened, SOLR-3779. This is also similar to | SOLR-3307, although that was reported as a bug with threads in | 3.6, which is no longer a feature in 4.0. | | James Dyer | E-Commerce Systems | Ingram Content Group | (615) 213-4311 | | | -Original Message- | From: Ahmet Arslan [mailto:iori...@yahoo.com] | Sent: Friday, August 31, 2012 1:53 PM | To: solr-user@lucene.apache.org | Subject: LineEntityProcessor process only one file | | LineEntityProcessor processes only one document when combined with | FileListEntityProcessor. Is this by design? | | | | |
Re: stem porter with tokenizer..
If you want to know class names, you want to check out the source code! http://lucene.apache.org/core/developer.html http://find.searchhub.org/s:javadoc,wiki?q=porterStemmer - Original Message - | From: Emiliana Suci emily_elz...@yahoo.com | To: solr-user@lucene.apache.org | Sent: Sunday, September 2, 2012 12:22:05 AM | Subject: stem porter with tokenizer.. | | PorterStemmer using tokenizer which class in lucene?? | | | | -- | View this message in context: | http://lucene.472066.n3.nabble.com/stem-porter-with-tokenizer-tp4004913.html | Sent from the Solr - User mailing list archive at Nabble.com. |
Re: Solr Not releasing memory
1) I believe Java 1.7 release memory back to the OS. 2) All of the Javas I've used on Windows do this. Is the physical memory use a problem? Does it push out all other programs? Or is it just that the Java process appears larger? This explains the latter: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html - Original Message - | From: Rohit ro...@simplify360.com | To: solr-user@lucene.apache.org | Sent: Sunday, September 2, 2012 1:22:14 AM | Subject: Solr Not releasing memory | | Hi, | | | | We are running solr3.5 using tomcal 6.26 on a Windows Enterprise RC2 | server, our index size if pretty large. | | | | We have noticed that once tomcat starts using/reserving ram it never | releases them, even when there is not a single user on the system. I | have | tried forced garbage collection, but that doesn't seem to help | either. | | | | Regards, | | Rohit | | | |
Antwort: Re: Antwort: Re: Query during a query
The problem is, that I don't know how to do this. :P My sequence: the user enters his search words. This is sent to solr. There I need to make another query first to get metadata from the index. with this metadata I have to connect to an external source to get some information about the user. With this information and the first search words I query then the solr index to get the search result. I hope its clear now wheres my problem and what I want to do Regards, Johannes Von: Jack Krupansky j...@basetechnology.com An: solr-user@lucene.apache.org Datum: 31.08.2012 15:03 Betreff: Re: Antwort: Re: Query during a query So, just do another query before doing the main query. What's the problem? Be more specific. Walk us through the sequence of processing that you need. -- Jack Krupansky -Original Message- From: johannes.schwendin...@blum.com Sent: Friday, August 31, 2012 1:52 AM To: solr-user@lucene.apache.org Subject: Antwort: Re: Query during a query Thanks for the answer, but I want to know how I can do a seperate query before the main query. And I only want this data in my programm. The user won't see it. I need the values from one field to get some information from an external source while the main query is executed. pravesh suyalprav...@yahoo.com schrieb am 31.08.2012 07:42:48: Von: pravesh suyalprav...@yahoo.com An: solr-user@lucene.apache.org Datum: 31.08.2012 07:43 Betreff: Re: Query during a query Did you checked SOLR Field Collapsing/Grouping. http://wiki.apache.org/solr/FieldCollapsing http://wiki.apache.org/solr/FieldCollapsing If this is what you are looking for. Thanx Pravesh -- View this message in context: http://lucene.472066.n3.nabble.com/ Query-during-a-query-tp4004624p4004631.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: need basic information
Do logstash/graylog2 do log processing/searching in real time? Or can scale for real time need? I guess harshadmehta is looking for real-time indexing/search. Regards Pravesh -- View this message in context: http://lucene.472066.n3.nabble.com/need-basic-information-tp4004588p4004996.html Sent from the Solr - User mailing list archive at Nabble.com.
