Re: Snapinstaller vs Solr Restart
Hi Otis, Ok about that, but still when it merges segments it changes names and I've no choice to replicate all the segment which is bad for the replication and cpu. ?? Thanks Otis Gospodnetic wrote: Lower your mergeFactor and Lucene will merge segments(i.e. fewer index files) and purge deletes more often for you at the expense of somewhat slower indexing. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: wojtekpia wojte...@hotmail.com To: solr-user@lucene.apache.org Sent: Tuesday, January 6, 2009 5:18:26 PM Subject: Re: Snapinstaller vs Solr Restart I'm optimizing because I thought I should. I'll be updating my index somewhere between every 15 minutes, and every 2 hours. That means between 12 and 96 updates per day. That seems like a lot of index files (and it scared me a little), so that's my second reason for wanting to optimize nightly. I haven't benchmarked the performance hit for not optimizing. That'll be my next step. If the hit isn't too bad, I'll look into optimizing less frequently (weekly, ...). Thanks Otis! Otis Gospodnetic wrote: OK, so that question/answer seems to have hit the nail on the head. :) When you optimize your index, all index files get rewritten. This means that everything that the OS cached up to that point goes out the window and the OS has to slowly re-cache the hot parts of the index. If you don't optimize, this won't happen. Do you really need to optimize? Or maybe a more direct question: why are you optimizing? Regarding autowarming, with such high fq hit rate, I'd make good use of fq autowarming. The result cache rate is lower, but still decent. I wouldn't turn off autowarming the way you have. -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21320334.html Sent from the Solr - User mailing list archive at Nabble.com. -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p22972780.html Sent from the Solr - User mailing list archive at Nabble.com.
Snapinstaller vs Solr Restart
I'm running load tests against my Solr instance. I find that it typically takes ~10 minutes for my Solr setup to warm-up while I throw my test queries at it. Also, I have the same two warm-up queries specified for the firstSearcher and newSearcher event listeners. I'm now benchmarking the affect of updating an index under load. I'm finding that after running snapinstaller, Solr takes ~1 hour to get back to the same performance numbers I was getting 10 minutes after a restart. If I can justify being offline for a few moments, it seems like I'll be better off restarting Solr rather than running Snapinstaller. Any ideas why? Thanks. -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21315273.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Snapinstaller vs Solr Restart
First suspect would be Filter Cache settings and Query Cache settings. If they are auto-warming at all, then there is a definite difference between the first start behavior and the post-commit behavior. This affects what's in memory, caches, etc. -Todd Feak -Original Message- From: wojtekpia [mailto:wojte...@hotmail.com] Sent: Tuesday, January 06, 2009 9:46 AM To: solr-user@lucene.apache.org Subject: Snapinstaller vs Solr Restart I'm running load tests against my Solr instance. I find that it typically takes ~10 minutes for my Solr setup to warm-up while I throw my test queries at it. Also, I have the same two warm-up queries specified for the firstSearcher and newSearcher event listeners. I'm now benchmarking the affect of updating an index under load. I'm finding that after running snapinstaller, Solr takes ~1 hour to get back to the same performance numbers I was getting 10 minutes after a restart. If I can justify being offline for a few moments, it seems like I'll be better off restarting Solr rather than running Snapinstaller. Any ideas why? Thanks. -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21315273. html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Snapinstaller vs Solr Restart
Sorry, I forgot to include that. All my autowarmcount's are set to 0. Feak, Todd wrote: First suspect would be Filter Cache settings and Query Cache settings. If they are auto-warming at all, then there is a definite difference between the first start behavior and the post-commit behavior. This affects what's in memory, caches, etc. -Todd Feak -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21315654.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Snapinstaller vs Solr Restart
Is autowarm count of 0 a good idea, though? If you don't want to autowarm any caches, doesn't that imply that you have very low hit rate and therefore don't care to autowarm? And if you have a very low hit rate, then perhaps caches are not needed at all? How about this. Do you optimize your index at any point? Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: wojtekpia wojte...@hotmail.com To: solr-user@lucene.apache.org Sent: Tuesday, January 6, 2009 1:07:28 PM Subject: RE: Snapinstaller vs Solr Restart Sorry, I forgot to include that. All my autowarmcount's are set to 0. Feak, Todd wrote: First suspect would be Filter Cache settings and Query Cache settings. If they are auto-warming at all, then there is a definite difference between the first start behavior and the post-commit behavior. This affects what's in memory, caches, etc. -Todd Feak -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21315654.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Snapinstaller vs Solr Restart
I use my warm up queries to fill the field cache (or at least that's the idea). My filterCache hit rate is ~99% queryResultCache is ~65%. I update my index several times a day with no 'optimize', and performance is seemless. I also update my index once nightly with an 'optimize', and that's where I see the performance drop. I'll try turning autowarming on. Could this have to do with file caching by the OS? Otis Gospodnetic wrote: Is autowarm count of 0 a good idea, though? If you don't want to autowarm any caches, doesn't that imply that you have very low hit rate and therefore don't care to autowarm? And if you have a very low hit rate, then perhaps caches are not needed at all? How about this. Do you optimize your index at any point? -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21319344.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Snapinstaller vs Solr Restart
OK, so that question/answer seems to have hit the nail on the head. :) When you optimize your index, all index files get rewritten. This means that everything that the OS cached up to that point goes out the window and the OS has to slowly re-cache the hot parts of the index. If you don't optimize, this won't happen. Do you really need to optimize? Or maybe a more direct question: why are you optimizing? Regarding autowarming, with such high fq hit rate, I'd make good use of fq autowarming. The result cache rate is lower, but still decent. I wouldn't turn off autowarming the way you have. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: wojtekpia wojte...@hotmail.com To: solr-user@lucene.apache.org Sent: Tuesday, January 6, 2009 4:20:18 PM Subject: Re: Snapinstaller vs Solr Restart I use my warm up queries to fill the field cache (or at least that's the idea). My filterCache hit rate is ~99% queryResultCache is ~65%. I update my index several times a day with no 'optimize', and performance is seemless. I also update my index once nightly with an 'optimize', and that's where I see the performance drop. I'll try turning autowarming on. Could this have to do with file caching by the OS? Otis Gospodnetic wrote: Is autowarm count of 0 a good idea, though? If you don't want to autowarm any caches, doesn't that imply that you have very low hit rate and therefore don't care to autowarm? And if you have a very low hit rate, then perhaps caches are not needed at all? How about this. Do you optimize your index at any point? -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21319344.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: Snapinstaller vs Solr Restart
Kind of a side-note, but I think it may be worth your while. If your queryResultCache hit rate is 65%, consider putting a reverse proxy in front of Solr. It can give performance boosts over the query cache in Solr, as it doesn't have to pay the cost of reformulating the response. I've used Varnish with great results. Squid is another option. -Todd Feak -Original Message- From: wojtekpia [mailto:wojte...@hotmail.com] Sent: Tuesday, January 06, 2009 1:20 PM To: solr-user@lucene.apache.org Subject: Re: Snapinstaller vs Solr Restart I use my warm up queries to fill the field cache (or at least that's the idea). My filterCache hit rate is ~99% queryResultCache is ~65%. I update my index several times a day with no 'optimize', and performance is seemless. I also update my index once nightly with an 'optimize', and that's where I see the performance drop. I'll try turning autowarming on. Could this have to do with file caching by the OS? Otis Gospodnetic wrote: Is autowarm count of 0 a good idea, though? If you don't want to autowarm any caches, doesn't that imply that you have very low hit rate and therefore don't care to autowarm? And if you have a very low hit rate, then perhaps caches are not needed at all? How about this. Do you optimize your index at any point? -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21319344. html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Snapinstaller vs Solr Restart
I'm optimizing because I thought I should. I'll be updating my index somewhere between every 15 minutes, and every 2 hours. That means between 12 and 96 updates per day. That seems like a lot of index files (and it scared me a little), so that's my second reason for wanting to optimize nightly. I haven't benchmarked the performance hit for not optimizing. That'll be my next step. If the hit isn't too bad, I'll look into optimizing less frequently (weekly, ...). Thanks Otis! Otis Gospodnetic wrote: OK, so that question/answer seems to have hit the nail on the head. :) When you optimize your index, all index files get rewritten. This means that everything that the OS cached up to that point goes out the window and the OS has to slowly re-cache the hot parts of the index. If you don't optimize, this won't happen. Do you really need to optimize? Or maybe a more direct question: why are you optimizing? Regarding autowarming, with such high fq hit rate, I'd make good use of fq autowarming. The result cache rate is lower, but still decent. I wouldn't turn off autowarming the way you have. -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21320334.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Snapinstaller vs Solr Restart
Lower your mergeFactor and Lucene will merge segments(i.e. fewer index files) and purge deletes more often for you at the expense of somewhat slower indexing. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: wojtekpia wojte...@hotmail.com To: solr-user@lucene.apache.org Sent: Tuesday, January 6, 2009 5:18:26 PM Subject: Re: Snapinstaller vs Solr Restart I'm optimizing because I thought I should. I'll be updating my index somewhere between every 15 minutes, and every 2 hours. That means between 12 and 96 updates per day. That seems like a lot of index files (and it scared me a little), so that's my second reason for wanting to optimize nightly. I haven't benchmarked the performance hit for not optimizing. That'll be my next step. If the hit isn't too bad, I'll look into optimizing less frequently (weekly, ...). Thanks Otis! Otis Gospodnetic wrote: OK, so that question/answer seems to have hit the nail on the head. :) When you optimize your index, all index files get rewritten. This means that everything that the OS cached up to that point goes out the window and the OS has to slowly re-cache the hot parts of the index. If you don't optimize, this won't happen. Do you really need to optimize? Or maybe a more direct question: why are you optimizing? Regarding autowarming, with such high fq hit rate, I'd make good use of fq autowarming. The result cache rate is lower, but still decent. I wouldn't turn off autowarming the way you have. -- View this message in context: http://www.nabble.com/Snapinstaller-vs-Solr-Restart-tp21315273p21320334.html Sent from the Solr - User mailing list archive at Nabble.com.
