Re: Custom close to index metadata / pass commit data to writer.commit
Hi, I just wanted to check if someone have an idea about intentions with this issue: https://issues.apache.org/jira/browse/SOLR-2701 It is marked for 4.0-Alpha and there is already Beta out there. Can anyone tell if it planed to be part of 4.0 release. Best, Jozef On Sun, Jun 24, 2012 at 1:18 AM, Erick Erickson wrote: > see: https://issues.apache.org/jira/browse/SOLR-2701. > > But there's an easier alternative. Just have a _very special_ document > with a known that you index at the end of the run that > 1> has no fields in common with any other document (except uniqueKey) > 2> contains whatever data you want to carry around in whatever format you > want. > > Now whenever you query for that document by ID, you get your info. And > since you can't search the doc until after it's been committed, you know > that the preceding documents have all been persisted > > Of course whenever you send a version of the doc it will overwrite the > one before since it has the same > > Best > Erick > > On Fri, Jun 22, 2012 at 5:34 AM, Jozef Vilcek wrote: >> Hi everyone, >> >> I am seeking to solution to store some custom data very close to / >> within index. I have found a possibility to pass commit "user" data to >> IndexWriter: >> http://lucene.apache.org/core/3_6_0/api/all/org/apache/lucene/index/IndexWriter.html#commit(java.util.Map) >> which are from what I understand stored somewhere close to segments >> "metadata" like index version, generation, ... >> >> Now, I see no easy way to accumulate and pass along such data with >> Solr 3.6. DirectUpdateHandler2 is committing implicitly via close >> rather than invoking commit API. I can extend DirectUpdateHander2 and >> alter closeWriter method but still ... I am not yet clear how to pass >> along request level params which are not available at >> DirectUpdateHandler2 level. It seems that passing commitData is not >> supported ( maybe not wanted to by by design ) and not going to be as >> when I look at Solr trunk, I see implicit commit removed, >> writer.commit with passing commitData used but no easy way how to pass >> custom commit data nor how to easily hook in. >> >> Any recommendations for how to store some data close to index? >> >> To throw some light why I what this ... Basically I want to store >> there some kind of time stamp, which defines what is already in the >> index with respect to feeding updates from external world. Now, my >> index is replicated to other index instance in different data center >> (serving traffic as well). When default document feed in DC1 go south >> for some reason, backup in DC2 bumps in to keep updates alive ... but >> it has to know from where the feed should start ... that would be that >> kind of time stamp stored and replicated with index. >> >> Many thanks in advance. >> >> Best, >> Jozef
Re: Custom close to index metadata / pass commit data to writer.commit
On Sun, Jun 24, 2012 at 1:18 AM, Erick Erickson wrote: > see: https://issues.apache.org/jira/browse/SOLR-2701. > Hey, that is what I want :) Thanks for the reference. I am unlucky that there seems to be no progress on this ( as far as I can tell ). I would be able to use commitData in rather non-invasive way in 3.6 release, but I fear of future and other releases ... > But there's an easier alternative. Just have a _very special_ document > with a known that you index at the end of the run that > 1> has no fields in common with any other document (except uniqueKey) > 2> contains whatever data you want to carry around in whatever format you > want. > > Now whenever you query for that document by ID, you get your info. And > since you can't search the doc until after it's been committed, you know > that the preceding documents have all been persisted > > Of course whenever you send a version of the doc it will overwrite the > one before since it has the same > Yes, we thought about having this data stored like a "special" document type, but conceptually it just does not feel right. Also, I fear of extra modifications and maintaining some query defaults to never return this document for any kind of search query ... > Best > Erick > > On Fri, Jun 22, 2012 at 5:34 AM, Jozef Vilcek wrote: >> Hi everyone, >> >> I am seeking to solution to store some custom data very close to / >> within index. I have found a possibility to pass commit "user" data to >> IndexWriter: >> http://lucene.apache.org/core/3_6_0/api/all/org/apache/lucene/index/IndexWriter.html#commit(java.util.Map) >> which are from what I understand stored somewhere close to segments >> "metadata" like index version, generation, ... >> >> Now, I see no easy way to accumulate and pass along such data with >> Solr 3.6. DirectUpdateHandler2 is committing implicitly via close >> rather than invoking commit API. I can extend DirectUpdateHander2 and >> alter closeWriter method but still ... I am not yet clear how to pass >> along request level params which are not available at >> DirectUpdateHandler2 level. It seems that passing commitData is not >> supported ( maybe not wanted to by by design ) and not going to be as >> when I look at Solr trunk, I see implicit commit removed, >> writer.commit with passing commitData used but no easy way how to pass >> custom commit data nor how to easily hook in. >> >> Any recommendations for how to store some data close to index? >> >> To throw some light why I what this ... Basically I want to store >> there some kind of time stamp, which defines what is already in the >> index with respect to feeding updates from external world. Now, my >> index is replicated to other index instance in different data center >> (serving traffic as well). When default document feed in DC1 go south >> for some reason, backup in DC2 bumps in to keep updates alive ... but >> it has to know from where the feed should start ... that would be that >> kind of time stamp stored and replicated with index. >> >> Many thanks in advance. >> >> Best, >> Jozef
Custom close to index metadata / pass commit data to writer.commit
Hi everyone, I am seeking to solution to store some custom data very close to / within index. I have found a possibility to pass commit "user" data to IndexWriter: http://lucene.apache.org/core/3_6_0/api/all/org/apache/lucene/index/IndexWriter.html#commit(java.util.Map) which are from what I understand stored somewhere close to segments "metadata" like index version, generation, ... Now, I see no easy way to accumulate and pass along such data with Solr 3.6. DirectUpdateHandler2 is committing implicitly via close rather than invoking commit API. I can extend DirectUpdateHander2 and alter closeWriter method but still ... I am not yet clear how to pass along request level params which are not available at DirectUpdateHandler2 level. It seems that passing commitData is not supported ( maybe not wanted to by by design ) and not going to be as when I look at Solr trunk, I see implicit commit removed, writer.commit with passing commitData used but no easy way how to pass custom commit data nor how to easily hook in. Any recommendations for how to store some data close to index? To throw some light why I what this ... Basically I want to store there some kind of time stamp, which defines what is already in the index with respect to feeding updates from external world. Now, my index is replicated to other index instance in different data center (serving traffic as well). When default document feed in DC1 go south for some reason, backup in DC2 bumps in to keep updates alive ... but it has to know from where the feed should start ... that would be that kind of time stamp stored and replicated with index. Many thanks in advance. Best, Jozef
How to handle filter query against empty fields
Hi all, I have a field(s) in a schema which I need to be able to specify in a filter query. The field is not mandatory, therefore it can be empty. I need to be able to run a query with a filer : " return only docs which does not have value for the field " ... What would be the optimal recommended way of doing this with Solr? Thanks! Best, Jozef
Binary updates handler does not propagate failures?
Hi all, I am facing following issue ... I have an application which is feeding Solr 3.6 index with document updates via Solrj 3.6. I use a binary request writer, because of the issue with XML when sending insert and deletes at once ( https://issues.apache.org/jira/browse/SOLR-1752 ) Now, I have noticed that if I sent a malformed document to the index, I see in logs it got refused by the index, but on the Solrj side, returned UpdateResponse does not indicate any kind of failure (no exception thrown, response status code == 0 ). When I switch to XML requests, I receive exception when sending malformed document. By looking at Solr's BinaryUpdateRequestHandler.java http://svn.apache.org/viewvc/lucene/dev/tags/lucene_solr_3_6_0/solr/core/src/java/org/apache/solr/handler/BinaryUpdateRequestHandler.java?view=markup at lines 98 - 105, exceptions are not propagated, therefore RequestHandlerBase can not set it into the response ... Is this intended behavior? What am I doing wrong? Any suggestions? Many thanks in advance. Best, Jozef
