Re: Can index size increase when no updates/optimizes are happening?
This is what we do, A user logs in - enter s a few documents in a particular domain, say A, B or C - logs out. Say B is the most commonly used domain. The increase in index size is drastic only in this particular domain. So unless a user logs in there s no question of documents being submitted or any indexing activity. From the logs i see no user logged in during that time frame. Adding to this we have to take a back up of the index every day and that s how we even came to know that such a problem even existed. Is there a possibility that i could schedule an optimize to run at a specific time during the day and hence try control the index s file size? On Sat, Mar 16, 2013 at 6:51 PM, Erick Erickson [via Lucene] ml-node+s472066n4047962...@n3.nabble.com wrote: Well, if nothing is going on at all, it's hard to see why the index would increase. So I suspect _something_ is going on. Possibilities: 1 you have indexing activity going on. Even if it's just replacing docs that already exist, which is actually an add and a delete the index will grow for a while; the deleted info isn't removed from the index until the segment is merged, which happens unpredictably (well, actually predictably but not on a schedule you enforce). So the index would grow/shrink/grow/shrink. Do you have automatic process in the background that push docs to the index? 2 You forceMerge (optimize), in which case the index will at least double in size temporarily. 3 You are replicating. While the replication goes on, especially if your index has changed greatly, then your index could double. None of these fit the symptoms you describe very well mind you. It's suspicious that these increases last long enough for you to see them predictably in the morning, unless, say, a background process indexes things regularly and you always look at the same time... unlikely at best. But the fact that your index goes from 18G to 8G strongly suggests that you're doing a forceMerge/optimize when you see it bump up to 18G. Best Erick On Sat, Mar 16, 2013 at 4:52 AM, eanand333 [hidden email]http://user/SendEmail.jtp?type=nodenode=4047962i=0 wrote: Hi, I am kind of new in here. Got the same question... I am using Java version 1.6 and Lucene version 3.3. Can the index file size increase automatically over night? During the evening i see the size around 11GB, next day morning i see it to be 18GB and again the size reduces around 8GB. I have checked the logs and i am sure that there was no user activity during this period. Mr. Eric has suggested this to be a user error, if that s true, i would like to know what are the possible errors which could result in the rapid increase in index file size ? Or what are the other possibilities for the index size to increase exponentially? Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Can-index-size-increase-when-no-updates-optimizes-are-happening-tp3334022p4047945.html Sent from the Solr - User mailing list archive at Nabble.com. -- If you reply to this email, your message will be added to the discussion below: http://lucene.472066.n3.nabble.com/Can-index-size-increase-when-no-updates-optimizes-are-happening-tp3334022p4047962.html To unsubscribe from Can index size increase when no updates/optimizes are happening?, click herehttp://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_codenode=3334022code=ZWFuYW5kLjA2OUBnbWFpbC5jb218MzMzNDAyMnwtNTIyMjg4NDE1 . NAMLhttp://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewerid=instant_html%21nabble%3Aemail.namlbase=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespacebreadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml -- View this message in context: http://lucene.472066.n3.nabble.com/Can-index-size-increase-when-no-updates-optimizes-are-happening-tp3334022p4048444.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Can index size increase when no updates/optimizes are happening?
Hi, I am kind of new in here. Got the same question... I am using Java version 1.6 and Lucene version 3.3. Can the index file size increase automatically over night? During the evening i see the size around 11GB, next day morning i see it to be 18GB and again the size reduces around 8GB. I have checked the logs and i am sure that there was no user activity during this period. Mr. Eric has suggested this to be a user error, if that s true, i would like to know what are the possible errors which could result in the rapid increase in index file size ? Or what are the other possibilities for the index size to increase exponentially? Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Can-index-size-increase-when-no-updates-optimizes-are-happening-tp3334022p4047945.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Can index size increase when no updates/optimizes are happening?
Well, if nothing is going on at all, it's hard to see why the index would increase. So I suspect _something_ is going on. Possibilities: 1 you have indexing activity going on. Even if it's just replacing docs that already exist, which is actually an add and a delete the index will grow for a while; the deleted info isn't removed from the index until the segment is merged, which happens unpredictably (well, actually predictably but not on a schedule you enforce). So the index would grow/shrink/grow/shrink. Do you have automatic process in the background that push docs to the index? 2 You forceMerge (optimize), in which case the index will at least double in size temporarily. 3 You are replicating. While the replication goes on, especially if your index has changed greatly, then your index could double. None of these fit the symptoms you describe very well mind you. It's suspicious that these increases last long enough for you to see them predictably in the morning, unless, say, a background process indexes things regularly and you always look at the same time... unlikely at best. But the fact that your index goes from 18G to 8G strongly suggests that you're doing a forceMerge/optimize when you see it bump up to 18G. Best Erick On Sat, Mar 16, 2013 at 4:52 AM, eanand333 eanand@gmail.com wrote: Hi, I am kind of new in here. Got the same question... I am using Java version 1.6 and Lucene version 3.3. Can the index file size increase automatically over night? During the evening i see the size around 11GB, next day morning i see it to be 18GB and again the size reduces around 8GB. I have checked the logs and i am sure that there was no user activity during this period. Mr. Eric has suggested this to be a user error, if that s true, i would like to know what are the possible errors which could result in the rapid increase in index file size ? Or what are the other possibilities for the index size to increase exponentially? Thanks -- View this message in context: http://lucene.472066.n3.nabble.com/Can-index-size-increase-when-no-updates-optimizes-are-happening-tp3334022p4047945.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Can index size increase when no updates/optimizes are happening?
I'd guess user error in the absence of stronger information. No docs added, no optimized kicked off, no replication and I can't think of any reason the index size should increase. Erick 2011/9/15 Yury Kats yuryk...@yahoo.com: On 9/14/2011 2:36 PM, Erick Erickson wrote: What is the machine used for? Was your user looking at a master? Slave? Something used for both? Stand-alone machine with multiple Solr cores. No replication. Measuring the size of all the files in the index? Or looking at memory? Disk space. The index files shouldn't be getting bigger unless there were indexing operations going on. That's what I thought. Is it at all possible that DIH was configured to run automatically (or any other indexing job for that matter) and your user didn't realize it? There's no DIH, but there is a custom app that submit docs for indexing via SolrJ. Supposedly, Solr logs were not showing any updates over night, so the assumption was that no new docs were added. I'd write it off as a user error, but wanted to double check with the community that no other internal Solr/Lucene task can change the index file size in the absence of submits.
Re: Can index size increase when no updates/optimizes are happening?
On 9/14/2011 2:36 PM, Erick Erickson wrote: What is the machine used for? Was your user looking at a master? Slave? Something used for both? Stand-alone machine with multiple Solr cores. No replication. Measuring the size of all the files in the index? Or looking at memory? Disk space. The index files shouldn't be getting bigger unless there were indexing operations going on. That's what I thought. Is it at all possible that DIH was configured to run automatically (or any other indexing job for that matter) and your user didn't realize it? There's no DIH, but there is a custom app that submit docs for indexing via SolrJ. Supposedly, Solr logs were not showing any updates over night, so the assumption was that no new docs were added. I'd write it off as a user error, but wanted to double check with the community that no other internal Solr/Lucene task can change the index file size in the absence of submits.
Re: Can index size increase when no updates/optimizes are happening?
What is the machine used for? Was your user looking at a master? Slave? Something used for both? Measuring the size of all the files in the index? Or looking at memory? The index files shouldn't be getting bigger unless there were indexing operations going on. Is it at all possible that DIH was configured to run automatically (or any other indexing job for that matter) and your user didn't realize it? Best Erick 2011/9/13 Yury Kats yuryk...@yahoo.com: One of my users observed that the index size (in bytes) increased over night. There was no indexing activity at that time, only querying was taking place. Running optimize brought the index size back down to what it was when indexing finished the day before. What could explain that?
Can index size increase when no updates/optimizes are happening?
One of my users observed that the index size (in bytes) increased over night. There was no indexing activity at that time, only querying was taking place. Running optimize brought the index size back down to what it was when indexing finished the day before. What could explain that?
