Re: Deleting an individual document while delta index is running
Okay, thanks for the help guys... I *think* that this can be resolved by kicking off the delta and passing optimize=false since the default was true in 3.3. I'll post back if I see the issue pop back up. JT On Wednesday, November 7, 2012 at 1:34 PM, Josh Turmel wrote: > Here's what we have set in our data-config.xml > > url="jdbc:postgresql://localhost:5432/reader" user="data" > batchSize="1000" readOnly="true" autoCommit="false" > transactionIsolation="TRANSACTION_READ_COMMITTED" > holdability="CLOSE_CURSORS_AT_COMMIT" > /> > > > Thanks, > Josh Turmel > > > On Wednesday, November 7, 2012 at 1:00 PM, Shawn Heisey wrote: > > > On 11/7/2012 10:55 AM, Otis Gospodnetic wrote: > > > Hi Shawn, > > > > > > It the last part really correct? Optimization should be doable while > > > updates are going on... or am I missing something? > > > > > > > > > From what I recall when I was first putting my build system together, > > which I will admit was on Solr 1.4.0, I couldn't do updates/deletes > > while optimizing was underway. I don't think 3.x was a whole lot > > different in this respect. From the little I understand about the > > significant changes in 4.0, it is probably now possible to do everything > > at the same time with no worry. > > > > Because they are using 3.3, I don't think they have access to this > > ability. Given the limited amount of information available, it seemed > > the most likely explanation. I could be wrong, and if I am, they will > > have to keep looking for an explanation. > > > > I'm definitely no expert, and I have not tried optimizing and updating > > at the same time since upgrading to 3.x. My indexing system > > deliberately avoids doing the two at the same time because it caused > > problems on 1.4.x. > > > > I would certainly love to know for sure whether it's possible on 4.0, > > because I am in the process of updating my entire test environment in > > preparation for a production rollout. If I can do updates/commits at > > the same time as optimizing, my code will get smaller and a lot simpler. > > > > Thanks, > > Shawn > > > > > > > >
Re: Deleting an individual document while delta index is running
Here's what we have set in our data-config.xml Thanks, Josh Turmel On Wednesday, November 7, 2012 at 1:00 PM, Shawn Heisey wrote: > On 11/7/2012 10:55 AM, Otis Gospodnetic wrote: > > Hi Shawn, > > > > It the last part really correct? Optimization should be doable while > > updates are going on... or am I missing something? > > > > > From what I recall when I was first putting my build system together, > which I will admit was on Solr 1.4.0, I couldn't do updates/deletes > while optimizing was underway. I don't think 3.x was a whole lot > different in this respect. From the little I understand about the > significant changes in 4.0, it is probably now possible to do everything > at the same time with no worry. > > Because they are using 3.3, I don't think they have access to this > ability. Given the limited amount of information available, it seemed > the most likely explanation. I could be wrong, and if I am, they will > have to keep looking for an explanation. > > I'm definitely no expert, and I have not tried optimizing and updating > at the same time since upgrading to 3.x. My indexing system > deliberately avoids doing the two at the same time because it caused > problems on 1.4.x. > > I would certainly love to know for sure whether it's possible on 4.0, > because I am in the process of updating my entire test environment in > preparation for a production rollout. If I can do updates/commits at > the same time as optimizing, my code will get smaller and a lot simpler. > > Thanks, > Shawn > >
Re: Deleting an individual document while delta index is running
On 11/7/2012 10:55 AM, Otis Gospodnetic wrote: Hi Shawn, It the last part really correct? Optimization should be doable while updates are going on... or am I missing something? From what I recall when I was first putting my build system together, which I will admit was on Solr 1.4.0, I couldn't do updates/deletes while optimizing was underway. I don't think 3.x was a whole lot different in this respect. From the little I understand about the significant changes in 4.0, it is probably now possible to do everything at the same time with no worry. Because they are using 3.3, I don't think they have access to this ability. Given the limited amount of information available, it seemed the most likely explanation. I could be wrong, and if I am, they will have to keep looking for an explanation. I'm definitely no expert, and I have not tried optimizing and updating at the same time since upgrading to 3.x. My indexing system deliberately avoids doing the two at the same time because it caused problems on 1.4.x. I would certainly love to know for sure whether it's possible on 4.0, because I am in the process of updating my entire test environment in preparation for a production rollout. If I can do updates/commits at the same time as optimizing, my code will get smaller and a lot simpler. Thanks, Shawn
Re: Deleting an individual document while delta index is running
Hi Shawn, It the last part really correct? Optimization should be doable while updates are going on... or am I missing something? Otis -- Search Analytics - http://sematext.com/search-analytics/index.html Performance Monitoring - http://sematext.com/spm/index.html On Wed, Nov 7, 2012 at 12:17 PM, Shawn Heisey wrote: > On 11/6/2012 2:01 PM, Josh Turmel wrote: > >> Running Solr 3.3 >> >> We're running into issues where deleting individual documents (by ID) will >> timeout but it only seems to happen when our hourly delta index is being >> ran to pull in new documents, is there a way to work around this? >> > > Are you experiencing really long commit times with your delta indexing, or > are you running an optimize after the delta index? That's the only thing I > can think of that might cause problems like that. > > General recommendations: 1) Reduce commit times. Large autowarm counts on > the Solr caches will make commits very slow. Also, make sure you've got > enough RAM so your OS disk cache runs efficiently. 2) Don't optimize > often, unless the index is really really small and it can complete quickly. > On an index that never sees document deletes or re-indexes, optimizing is > completely unnecessary. Keep in mind that re-indexing an existing document > does an implicit delete first. Since you are deleting documents, you'll > want to do an optimize occasionally to keep your index clean, but you > should do it infrequently - once a day or similar. > > Another critical note about optimizing: No matter how often you do it, > you'll want to make sure that no other updates are happening on that index > while it is optimizing. Since I know this may be the obvious next > question: I am not aware of any way to query Solr and ask it if an optimize > is underway. You'll need to coordinate it yourself. > > Thanks, > Shawn > >
Re: Deleting an individual document while delta index is running
On 11/6/2012 2:01 PM, Josh Turmel wrote: Running Solr 3.3 We're running into issues where deleting individual documents (by ID) will timeout but it only seems to happen when our hourly delta index is being ran to pull in new documents, is there a way to work around this? Are you experiencing really long commit times with your delta indexing, or are you running an optimize after the delta index? That's the only thing I can think of that might cause problems like that. General recommendations: 1) Reduce commit times. Large autowarm counts on the Solr caches will make commits very slow. Also, make sure you've got enough RAM so your OS disk cache runs efficiently. 2) Don't optimize often, unless the index is really really small and it can complete quickly. On an index that never sees document deletes or re-indexes, optimizing is completely unnecessary. Keep in mind that re-indexing an existing document does an implicit delete first. Since you are deleting documents, you'll want to do an optimize occasionally to keep your index clean, but you should do it infrequently - once a day or similar. Another critical note about optimizing: No matter how often you do it, you'll want to make sure that no other updates are happening on that index while it is optimizing. Since I know this may be the obvious next question: I am not aware of any way to query Solr and ask it if an optimize is underway. You'll need to coordinate it yourself. Thanks, Shawn
Deleting an individual document while delta index is running
Running Solr 3.3 We're running into issues where deleting individual documents (by ID) will timeout but it only seems to happen when our hourly delta index is being ran to pull in new documents, is there a way to work around this? Thank you, Josh