deleteById commitWithin question
I am running off of a snapshot taken 5/3/2012 of solr 4.0 and am noticing some issues around deleteById when a commitWithin parameter is included using SolrJ, specifically commit isn't executed. If I later just call commit on the solr instance I see the item is deleted though. Is anyone aware if this should work in that snapshot?
Re: deleteById commitWithin question
On Thu, Jul 5, 2012 at 4:29 PM, Jamie Johnson wrote: > I am running off of a snapshot taken 5/3/2012 of solr 4.0 and am > noticing some issues around deleteById when a commitWithin parameter > is included using SolrJ, specifically commit isn't executed. If I > later just call commit on the solr instance I see the item is deleted > though. Is anyone aware if this should work in that snapshot? I thought I remembered something like this... but looking at the commit log for DUH2, I don't see it. /opt/code/lusolr4$ svn log ./solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java | less r1357332 | yonik | 2012-07-04 12:23:09 -0400 (Wed, 04 Jul 2012) | 1 line log DBQ reordering events r1356858 | markrmiller | 2012-07-03 14:18:48 -0400 (Tue, 03 Jul 2012) | 1 line SOLR-3587: After reloading a SolrCore, the original Analyzer is still used rather than a new one r1356845 | yonik | 2012-07-03 13:47:56 -0400 (Tue, 03 Jul 2012) | 1 line SOLR-3559: DBQ reorder support r1355088 | sarowe | 2012-06-28 13:51:38 -0400 (Thu, 28 Jun 2012) | 1 line LUCENE-4172: clean up redundant throws clauses (merge from trunk) r1348984 | hossman | 2012-06-11 15:46:14 -0400 (Mon, 11 Jun 2012) | 1 line LUCENE-3949: fix license headers to not be javadoc style comments r1343813 | rmuir | 2012-05-29 12:16:38 -0400 (Tue, 29 May 2012) | 1 line create stable branch for 4.x releases r1328890 | yonik | 2012-04-22 11:01:55 -0400 (Sun, 22 Apr 2012) | 1 line SOLR-3392: fix search leak when openSearcher=false r1328883 | yonik | 2012-04-22 09:58:00 -0400 (Sun, 22 Apr 2012) | 1 line SOLR-3391: Make explicit commits cancel pending autocommits. I'll try out trunk quick and see if it currently works. -Yonik http://lucidimagination.com
Re: deleteById commitWithin question
On Thu, Jul 5, 2012 at 4:29 PM, Jamie Johnson wrote: > I am running off of a snapshot taken 5/3/2012 of solr 4.0 and am > noticing some issues around deleteById when a commitWithin parameter > is included using SolrJ Oh wait... I just realized you were talking about SolrJ specifically - so the issue may be there. -Yonik http://lucidimagination.com
Re: deleteById commitWithin question
Thanks for the fast response. Yeah, it could be in either place either SolrJ or on the Server side. On Thu, Jul 5, 2012 at 4:47 PM, Yonik Seeley wrote: > On Thu, Jul 5, 2012 at 4:29 PM, Jamie Johnson wrote: >> I am running off of a snapshot taken 5/3/2012 of solr 4.0 and am >> noticing some issues around deleteById when a commitWithin parameter >> is included using SolrJ > > Oh wait... I just realized you were talking about SolrJ specifically - > so the issue may be there. > > -Yonik > http://lucidimagination.com
Re: deleteById commitWithin question
Ok, so some more context, hopefully this is useful. I didn't think this was a SolrCloud issue but it appears to be. I have a simple 2 shard set up, I add 1 document which goes to shard 1. I then issue a delete to shard 2. The delete gets there and I see the commit eventually being run in the log, but it doesn't appear to be being distributed to shard 1. If I manually call a server.commit() the item is removed. On Thu, Jul 5, 2012 at 4:56 PM, Jamie Johnson wrote: > Thanks for the fast response. Yeah, it could be in either place > either SolrJ or on the Server side. > > On Thu, Jul 5, 2012 at 4:47 PM, Yonik Seeley > wrote: >> On Thu, Jul 5, 2012 at 4:29 PM, Jamie Johnson wrote: >>> I am running off of a snapshot taken 5/3/2012 of solr 4.0 and am >>> noticing some issues around deleteById when a commitWithin parameter >>> is included using SolrJ >> >> Oh wait... I just realized you were talking about SolrJ specifically - >> so the issue may be there. >> >> -Yonik >> http://lucidimagination.com
Re: deleteById commitWithin question
On Thu, Jul 5, 2012 at 5:12 PM, Jamie Johnson wrote: > Ok, so some more context, hopefully this is useful. > > I didn't think this was a SolrCloud issue but it appears to be. I > have a simple 2 shard set up, I add 1 document which goes to shard 1. > I then issue a delete to shard 2. The delete gets there and I see the > commit eventually being run in the log, but it doesn't appear to be > being distributed to shard 1. If I manually call a server.commit() > the item is removed. OK, I just verified this for trunk / 4x. It appears to work fine now, but I'm not exactly sure why it may not with the snapshot you have (much has changed since). I followed the solr cloud wiki except that the path now needs to have collection1 in it, and I set the number of shards to 1 (with 2 replicas). I indexed the example docs, then queried both shards to verify the doc existed on both. Then I deleted the doc through the non-leader via: curl "http://localhost:7574/solr/update?commitWithin=6"; -H 'Content-type:application/json' -d '{"delete": { "id":"SOLR1000"}}' This was to test both the forwarding to the leader and the fowarding from the leader to the replica. After 60 seconds, the document was gone on both replicas. aside: If you use deleteByQuery, be aware that full distributed indexing support was just finished yesterday by SOLR-3559 -Yonik http://lucidimagination.com
Re: deleteById commitWithin question
Thanks Yonik, glad that it is not an issue on Trunk. I'll see if our group is interested in updating now to take advantage of this. In the mean time we just issue a commit after we do deletes and that seems to be a sufficient workaround. On Thu, Jul 5, 2012 at 6:05 PM, Yonik Seeley wrote: > On Thu, Jul 5, 2012 at 5:12 PM, Jamie Johnson wrote: >> Ok, so some more context, hopefully this is useful. >> >> I didn't think this was a SolrCloud issue but it appears to be. I >> have a simple 2 shard set up, I add 1 document which goes to shard 1. >> I then issue a delete to shard 2. The delete gets there and I see the >> commit eventually being run in the log, but it doesn't appear to be >> being distributed to shard 1. If I manually call a server.commit() >> the item is removed. > > OK, I just verified this for trunk / 4x. It appears to work fine now, > but I'm not exactly sure why it may not with the snapshot you have > (much has changed since). > > I followed the solr cloud wiki except that the path now needs to have > collection1 in it, and I set the number of shards to 1 (with 2 > replicas). I indexed the example docs, then queried both shards to > verify the doc existed on both. Then I deleted the doc through the > non-leader via: > curl "http://localhost:7574/solr/update?commitWithin=6"; -H > 'Content-type:application/json' -d '{"delete": { "id":"SOLR1000"}}' > > This was to test both the forwarding to the leader and the fowarding > from the leader to the replica. > After 60 seconds, the document was gone on both replicas. > > aside: If you use deleteByQuery, be aware that full distributed > indexing support was just finished yesterday by SOLR-3559 > > -Yonik > http://lucidimagination.com