Re: Solr + Tomcat Undeploy Leaks
On 10/18/07, Mike Klaas [EMAIL PROTECTED] wrote: I have a large number of servers, each running 1-2 containers, each having 1-2 solr deployments (fixed). If I want a new Solr instance, I just start a new container (possibly on a new server). I treat it like a process, and can shut it down using kill and other unix tools. I realize this is a bit off-topic -- but I'm curious what the rationale was behind having that many solr instances on that many machines and how they are coordinated. Is it a master/slave setup or are they distinct indexes? Any further details about your architecture would be interesting to read about :-) //Ed
Re: Solr + Tomcat Undeploy Leaks
On 19-Oct-07, at 7:19 AM, Ed Summers wrote: On 10/18/07, Mike Klaas [EMAIL PROTECTED] wrote: I realize this is a bit off-topic -- but I'm curious what the rationale was behind having that many solr instances on that many machines and how they are coordinated. Is it a master/slave setup or are they distinct indexes? Any further details about your architecture would be interesting to read about :-) Rationale? Performance! I can't divulge the exact size of our corpus, but it is between zero and 1 billion web documents. To search that many documents efficiently requires distributing over many machines. Most of the architecture is not Solr-related, but it is pretty standard large-scale search engine stuff (namely, distributing documents using some sort of unique hash across multiple machines). I'm sure Nutch's design is similar, and there are several academic papers on the subject. Solr plays the role of index at the nodes--it isn't the primary document storage. Each individual index doesn't look so different from a typical-size Solr index: the main differences are 1) splitting the stored fields among two Solr apps running in a single jvm for io performance (for highlighting) 2) scoring/query tweaks. cheers, -Mike
RE: Solr + Tomcat Undeploy Leaks
Any ideas? Has anyone had experienced this problem with other containers? I'm not tied to Tomcat if I can find another servlet host with a REST api for deploying apps. Thanks, Stu -Original Message- From: Stu Hood [EMAIL PROTECTED] Sent: Wednesday, October 17, 2007 4:46pm To: solr-user@lucene.apache.org Subject: Solr + Tomcat Undeploy Leaks Hello, I'm using the Tomcat Manager app with 6.0.14 to start and stop Solr instances, and I believe I am running into a variant of the linked issue: http://wiki.apache.org/jakarta-commons/Logging/UndeployMemoryLeak?action=print According to `top`, the 'size' of the Tomcat process reaches the limit I have set for it with the Java -Xmx flag soon after starting and launching a few instances. The 'RSS' varies based on how full the caches are at any particular time, but I don't think it ever reaches the 'size'. After a few days, I will get OOM errors in the logs when I try and start new instances (note: this is typically in the middle of the night, when usage is low), and all of the instances will stop responding until I (hard) restart Tomcat. Has anyone run into this issue before? Is logging the culprit? If so, what options do I have (besides setting up a cron job to restart Tomcat nightly...) Thanks, Stu Hood Webmail.us You manage your business. We'll manage your email.®
Re: Solr + Tomcat Undeploy Leaks
I'm not sure that many people are dynamically taking down/starting up Solr webapps in servlet containers. I certainly perfer process-level management of my (many) Solr instances. -Mike On 18-Oct-07, at 10:40 AM, Stu Hood wrote: Any ideas? Has anyone had experienced this problem with other containers? I'm not tied to Tomcat if I can find another servlet host with a REST api for deploying apps. Thanks, Stu -Original Message- From: Stu Hood [EMAIL PROTECTED] Sent: Wednesday, October 17, 2007 4:46pm To: solr-user@lucene.apache.org Subject: Solr + Tomcat Undeploy Leaks Hello, I'm using the Tomcat Manager app with 6.0.14 to start and stop Solr instances, and I believe I am running into a variant of the linked issue: http://wiki.apache.org/jakarta-commons/Logging/UndeployMemoryLeak? action=print According to `top`, the 'size' of the Tomcat process reaches the limit I have set for it with the Java -Xmx flag soon after starting and launching a few instances. The 'RSS' varies based on how full the caches are at any particular time, but I don't think it ever reaches the 'size'. After a few days, I will get OOM errors in the logs when I try and start new instances (note: this is typically in the middle of the night, when usage is low), and all of the instances will stop responding until I (hard) restart Tomcat. Has anyone run into this issue before? Is logging the culprit? If so, what options do I have (besides setting up a cron job to restart Tomcat nightly...) Thanks, Stu Hood Webmail.us You manage your business. We'll manage your email.®
Re: Solr + Tomcat Undeploy Leaks
I certainly have seen memory problems when I just drop a new war file in place. So now I usually stop tomcat and restart. I used to see problems (pre-1.0) when I just redeployed repeatedly, without even accessing the app, but I've got a little script running in the background that has done that 50 times now, without running out of space. Are you on a current version? I'm on 1.2 Tlom On 10/18/07, Mike Klaas [EMAIL PROTECTED] wrote: I'm not sure that many people are dynamically taking down/starting up Solr webapps in servlet containers. I certainly perfer process-level management of my (many) Solr instances. -Mike On 18-Oct-07, at 10:40 AM, Stu Hood wrote: Any ideas? Has anyone had experienced this problem with other containers? I'm not tied to Tomcat if I can find another servlet host with a REST api for deploying apps. Thanks, Stu -Original Message- From: Stu Hood [EMAIL PROTECTED] Sent: Wednesday, October 17, 2007 4:46pm To: solr-user@lucene.apache.org Subject: Solr + Tomcat Undeploy Leaks Hello, I'm using the Tomcat Manager app with 6.0.14 to start and stop Solr instances, and I believe I am running into a variant of the linked issue: http://wiki.apache.org/jakarta-commons/Logging/UndeployMemoryLeak? action=print According to `top`, the 'size' of the Tomcat process reaches the limit I have set for it with the Java -Xmx flag soon after starting and launching a few instances. The 'RSS' varies based on how full the caches are at any particular time, but I don't think it ever reaches the 'size'. After a few days, I will get OOM errors in the logs when I try and start new instances (note: this is typically in the middle of the night, when usage is low), and all of the instances will stop responding until I (hard) restart Tomcat. Has anyone run into this issue before? Is logging the culprit? If so, what options do I have (besides setting up a cron job to restart Tomcat nightly...) Thanks, Stu Hood Webmail.us You manage your business. We'll manage your email.(R)
Re: Solr + Tomcat Undeploy Leaks
On 18-Oct-07, at 1:01 PM, Stu Hood wrote: I'm running SVN r583865 (1.3-dev). Mike: when you say 'process level management', do you mean starting them statically? Or starting them dynamically, but using a different container for each instance? I have a large number of servers, each running 1-2 containers, each having 1-2 solr deployments (fixed). If I want a new Solr instance, I just start a new container (possibly on a new server). I treat it like a process, and can shut it down using kill and other unix tools. A little explanation is probably in order: -- We're using Solr to provide log search capability to our customer service department. We keep 7 days of logs available by starting up a new instance every 6 hours or so and stopping the oldest instance. We merge in pre-indexed logs every 10 minutes. Could you run one process/container per instance? We have a few nodes providing search, so we have one machine that schedules Solr instances on the nodes with Tomcat Manager, and all instances are searched simultaneously using the patch from SOLR-303. An option I'm considering is keeping a static number of instances on each node, and doing a id[* TO *] delete when an instance needs to be reused, but I'd rather figure this bug out than refactor the scheduling/merging code for static instances. Java ought to be able to GC its way out of situations like this... Note that deleting query*:*/query is optimized in trunk to be ridiculously fast (basically equivalent to just unlinking the files). -Mike
Re: Solr + Tomcat Undeploy Leaks
I'm running SVN r583865 (1.3-dev). Mike: when you say 'process level management', do you mean starting them statically? Or starting them dynamically, but using a different container for each instance? A little explanation is probably in order: -- We're using Solr to provide log search capability to our customer service department. We keep 7 days of logs available by starting up a new instance every 6 hours or so and stopping the oldest instance. We merge in pre-indexed logs every 10 minutes. We have a few nodes providing search, so we have one machine that schedules Solr instances on the nodes with Tomcat Manager, and all instances are searched simultaneously using the patch from SOLR-303. An option I'm considering is keeping a static number of instances on each node, and doing a id[* TO *] delete when an instance needs to be reused, but I'd rather figure this bug out than refactor the scheduling/merging code for static instances. Java ought to be able to GC its way out of situations like this... Thanks, Stu -Original Message- From: Tom Hill [EMAIL PROTECTED] Sent: Thursday, October 18, 2007 3:34pm To: solr-user@lucene.apache.org Subject: Re: Solr + Tomcat Undeploy Leaks I certainly have seen memory problems when I just drop a new war file in place. So now I usually stop tomcat and restart. I used to see problems (pre-1.0) when I just redeployed repeatedly, without even accessing the app, but I've got a little script running in the background that has done that 50 times now, without running out of space. Are you on a current version? I'm on 12 Tlom On 10/18/07, Mike Klaas [EMAIL PROTECTED] wrote: I'm not sure that many people are dynamically taking down/starting up Solr webapps in servlet containers. I certainly perfer process-level management of my (many) Solr instances. -Mike On 18-Oct-07, at 10:40 AM, Stu Hood wrote: Any ideas? Has anyone had experienced this problem with other containers? I'm not tied to Tomcat if I can find another servlet host with a REST api for deploying apps. Thanks, Stu -Original Message- From: Stu Hood [EMAIL PROTECTED] Sent: Wednesday, October 17, 2007 4:46pm To: solr-user@lucene.apache.org Subject: Solr + Tomcat Undeploy Leaks Hello, I'm using the Tomcat Manager app with 6.0.14 to start and stop Solr instances, and I believe I am running into a variant of the linked issue: http://wiki.apache.org/jakarta-commons/Logging/UndeployMemoryLeak? action=print According to `top`, the 'size' of the Tomcat process reaches the limit I have set for it with the Java -Xmx flag soon after starting and launching a few instances. The 'RSS' varies based on how full the caches are at any particular time, but I don't think it ever reaches the 'size'. After a few days, I will get OOM errors in the logs when I try and start new instances (note: this is typically in the middle of the night, when usage is low), and all of the instances will stop responding until I (hard) restart Tomcat. Has anyone run into this issue before? Is logging the culprit? If so, what options do I have (besides setting up a cron job to restart Tomcat nightly...) Thanks, Stu Hood Webmail.us You manage your business. We'll manage your email.(R)
Solr + Tomcat Undeploy Leaks
Hello, I'm using the Tomcat Manager app with 6.0.14 to start and stop Solr instances, and I believe I am running into a variant of the linked issue: http://wiki.apache.org/jakarta-commons/Logging/UndeployMemoryLeak?action=print According to `top`, the 'size' of the Tomcat process reaches the limit I have set for it with the Java -Xmx flag soon after starting and launching a few instances. The 'RSS' varies based on how full the caches are at any particular time, but I don't think it ever reaches the 'size'. After a few days, I will get OOM errors in the logs when I try and start new instances (note: this is typically in the middle of the night, when usage is low), and all of the instances will stop responding until I (hard) restart Tomcat. Has anyone run into this issue before? Is logging the culprit? If so, what options do I have (besides setting up a cron job to restart Tomcat nightly...) Thanks, Stu Hood Webmail.us You manage your business. We'll manage your email.®
