Re: Solr4.0 / SolrCloud queries
Hi all , I have managed to successfully index around 6 million documents, but while indexing (and even now after the indexing has stopped), I am running into a bunch of errors. The most common error I see is / null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://ABC:8983/solr/xyzabc/ I have made sure that the servers are able to communicate with each other using the same names. Another error I keep getting is that the leader stops recovering and goes red / recovery failed. /Error while trying to recover. core=ABC123:org.apache.solr.common.SolrException: We are not the leader/ The servers intermittently go offline taking down one of the shards and in turn stopping all search queries. The configuration I have Shard1: Server1 - Memory - 22GB , JVM - 8gb Server2 - Memory - 22GB , JVM - 10gb (This one is on recovery failed status, but still acting as a leader). Shard2: Server1 - Memory - 22GB , JVM - 8 GB (This one is on recovery failed status, but still acting as a leader). Server2 - Memory - 22 GB, JVM - 8 GB Shard3 Server1 - Memory - 22 GB, JVM - 10 GB Server2 - Memory - 22 GB, JVM - 8 GB While typing his post I did a Reload from the Core Admin page, and both servers (Shard1-Server2 and Shard2-Server1)came back up again. Has anyone else encountered these issues? Any steps to prevent these? Thanks. --Shreejay -- View this message in context: http://lucene.472066.n3.nabble.com/Solr4-0-SolrCloud-queries-tp4016825p4021154.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr4.0 / SolrCloud queries
Thanks Mark. I meant ConcurrentMergeScheduler and ramBufferSizeMB (not maxBuffer). These are my settings for Merge. / ramBufferSizeMB960/ramBufferSizeMB mergeFactor40/mergeFactor mergeScheduler class=org.apache.lucene.index.ConcurrentMergeScheduler/ / --Shreejay Mark Miller-3 wrote On Nov 9, 2012, at 1:20 PM, shreejay lt; shreejayn@ gt; wrote: Instead of doing an optimize, I have now changed the Merge settings by keeping a maxBuffer = 960, a merge Factor = 40 and ConcurrentMergePolicy. Don't you mean ConcurrentMergeScheduler? Keep in mind that if you use the default TieredMergePolicy, mergeFactor will have no affect. You need to use maxMergeAtOnce and segmentsPerTier as sub args to the merge policy config (see the commented out example in solrconfig.xml). Also, it's probably best to avoid using maxBufferedDocs at all. - Mark Mark Miller-3 wrote On Nov 9, 2012, at 1:20 PM, shreejay lt; shreejayn@ gt; wrote: Instead of doing an optimize, I have now changed the Merge settings by keeping a maxBuffer = 960, a merge Factor = 40 and ConcurrentMergePolicy. Don't you mean ConcurrentMergeScheduler? Keep in mind that if you use the default TieredMergePolicy, mergeFactor will have no affect. You need to use maxMergeAtOnce and segmentsPerTier as sub args to the merge policy config (see the commented out example in solrconfig.xml). Also, it's probably best to avoid using maxBufferedDocs at all. - Mark -- View this message in context: http://lucene.472066.n3.nabble.com/Solr4-0-SolrCloud-queries-tp4016825p4020200.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr4.0 / SolrCloud queries
You really should be careful about optimizes, they're generally not needed. And optimizing is almost always wrong when done after every N documents in a batch process. Do it at the very end or not at all. optimize essentially re-writes the entire index into a single segment, so you're copying around a lot of data. And the operations that get done during optimize, which are mainly purging information associated with deleted documents, gets done anyway upon segment merging. Despite it's name, optimize usually has marginal effect, if any, on search speed. Or did you mean commit? FWIW, Erick On Thu, Nov 8, 2012 at 4:05 PM, shreejay shreej...@gmail.com wrote: I managed to re-index my data without issues. I indexed around 2 million documents in one of the clouds. I did an optimize after every 500k documents. I also changed the Memory settings and assigned only 6gb for Java , and kept 10Gb for OS. This seems to be working fine as of now. I am not seeing random leader selections and servers drops. Thanks everyone for your inputs. --Shreejay -- View this message in context: http://lucene.472066.n3.nabble.com/Solr4-0-SolrCloud-queries-tp4016825p4019144.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr4.0 / SolrCloud queries
Thanks Erick. I will try optimizing after indexing everything. I was doing it after every batch since it was taking way too long to Optimize (which was expected), but it was not finishing merging it into lesser number of segments (1 segment). Instead of doing an optimize, I have now changed the Merge settings by keeping a maxBuffer = 960, a merge Factor = 40 and ConcurrentMergePolicy. I am also going to check the infoStream option so I can see how the indexing is going on. Thanks for your inputs. --Shreejay -- View this message in context: http://lucene.472066.n3.nabble.com/Solr4-0-SolrCloud-queries-tp4016825p4019373.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr4.0 / SolrCloud queries
On Nov 9, 2012, at 1:20 PM, shreejay shreej...@gmail.com wrote: Instead of doing an optimize, I have now changed the Merge settings by keeping a maxBuffer = 960, a merge Factor = 40 and ConcurrentMergePolicy. Don't you mean ConcurrentMergeScheduler? Keep in mind that if you use the default TieredMergePolicy, mergeFactor will have no affect. You need to use maxMergeAtOnce and segmentsPerTier as sub args to the merge policy config (see the commented out example in solrconfig.xml). Also, it's probably best to avoid using maxBufferedDocs at all. - Mark
Re: Solr4.0 / SolrCloud queries
Thanks Everyone. As Shawn mentioned, it was a memory issue. I reduced the amount allocated to Java to 6 GB. And its been working pretty good. I am re-indexing one of the SolrCloud. I was having trouble with optimizing the data when I indexed last time I am hoping optimizing will not be an issue this time due to the memory changes. I will post more info once I am done. Thanks once again. --Shreejay -- View this message in context: http://lucene.472066.n3.nabble.com/Solr4-0-SolrCloud-queries-tp4016825p4018176.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr4.0 / SolrCloud queries
If you can share any logs, that would help as well. - Mark
Re: Solr4.0 / SolrCloud queries
In addition to Shawn's comments, you might want to see: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html Lucene's use of MMapDirectory can mislead you when looking at memory usage Best Erick On Mon, Oct 29, 2012 at 5:59 PM, Shawn Heisey s...@elyograg.org wrote: On 10/29/2012 3:26 PM, shreejay wrote: I am trying to run two SolrCloud with 3 and 2 shards respectively (lets say Cloud3shards and Clouds2Shards). All servers are identical with 18GB Ram (16GB assigned for Java). This bit right here sets off warning bells right away. You're only leaving 2GB of RAM for the OS to cache your index, which you later say is 50GB. It's impossible for me to give you a precise figure, but I would expect that with an index that size, you'd want to have at least 20GB of free memory, and if you can have 50GB or more of free memory after the OS and Java take their chunk, Solr would have truly excellent performance. As it is now, your performance will be terrible, which probably explains all your issues. It seems highly unlikely that you would have queries complex enough that you actually do need to allocate 16GB of RAM to Java. Also, requesting large numbers of documents (the 5000 and 2 numbers you mentioned) is slow, and compounded in a cloud (distributed) index. Solr is optimized for a small number of results. First recommendation for fixing things: get more memory. 32GB would be a good starting point, 64GB would be better, so that the entire index will be able to fit in OS cache memory. If you expect your index to grow at all, plan accordingly. Second recommendation, whether or not you get more actual memory: Lower the memory that Java is using, and configure some alternate memory management options for Java. Solr does have caching capability, but it is highly specialized. For general index caching, the OS does a far better job, and it needs free memory in order to accomplish it. Here's some commandline options for Java that I passed along to someone else on this list: -Xmx4096M -Xms4096M -XX:NewRatio=1 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled http://www.petefreitag.com/articles/gctuning/
Solr4.0 / SolrCloud queries
Hi All, I am trying to run two SolrCloud with 3 and 2 shards respectively (lets say Cloud3shards and Clouds2Shards). All servers are identical with 18GB Ram (16GB assigned for Java). I am facing a few issues on both clouds and would be grateful if any one else has seen / solved these. 1) Every now and then, Solr would take off one of the servers (It either shows as recovering (orange) or its taken offline completely). The Logging tab on Admin page shows these errors for Cloud3shards /Error while trying to recover:org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://xxx:8983/solr/xxx / /Error while trying to recover. core=xxx:org.apache.solr.common.SolrException: I was asked to wait on state recovering for xxx:8983_solr but I still do not see the request state. I see state: recovering live:false/ On the Cloud2shards also I see similar messages I have noticed it does happen more while indexing documents, but I have also seen this happening while only querying Solr. Both SolrClouds are managed by the same Zookeeper ensemble (set of 3 ZK servers). 2) I am able to Commit but Optimize never seems to work. Right now I have an average of 30 segments on every Solr Server. Has any one else faced this issue? I have tried Optimize from the admin page and as a Http post request. Both of them fail. Its not because of the hard disk space since my index size is less than 50Gb and I have 500GB space on each server. 3) If I try to query solr with rows = 5000 or more (for Cloud2) . for cloud1 its around 20,000 documents. org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://ABC1:8983/solr/aaa, http://ABC2:8983/solr/aaa]. 4) I have also noticed that ZK would switch leaders every now and then. I am attributing it to point 1 above, where as soon as the leader is down, another server takes its place. My concern is the frequency with which this switch happens. I guess this is completely related to point 1 , and if that is solved, I will not be having this issue either. --Shreejay -- View this message in context: http://lucene.472066.n3.nabble.com/Solr4-0-SolrCloud-queries-tp4016825.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr4.0 / SolrCloud queries
On 10/29/2012 3:26 PM, shreejay wrote: I am trying to run two SolrCloud with 3 and 2 shards respectively (lets say Cloud3shards and Clouds2Shards). All servers are identical with 18GB Ram (16GB assigned for Java). This bit right here sets off warning bells right away. You're only leaving 2GB of RAM for the OS to cache your index, which you later say is 50GB. It's impossible for me to give you a precise figure, but I would expect that with an index that size, you'd want to have at least 20GB of free memory, and if you can have 50GB or more of free memory after the OS and Java take their chunk, Solr would have truly excellent performance. As it is now, your performance will be terrible, which probably explains all your issues. It seems highly unlikely that you would have queries complex enough that you actually do need to allocate 16GB of RAM to Java. Also, requesting large numbers of documents (the 5000 and 2 numbers you mentioned) is slow, and compounded in a cloud (distributed) index. Solr is optimized for a small number of results. First recommendation for fixing things: get more memory. 32GB would be a good starting point, 64GB would be better, so that the entire index will be able to fit in OS cache memory. If you expect your index to grow at all, plan accordingly. Second recommendation, whether or not you get more actual memory: Lower the memory that Java is using, and configure some alternate memory management options for Java. Solr does have caching capability, but it is highly specialized. For general index caching, the OS does a far better job, and it needs free memory in order to accomplish it. Here's some commandline options for Java that I passed along to someone else on this list: -Xmx4096M -Xms4096M -XX:NewRatio=1 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled http://www.petefreitag.com/articles/gctuning/
