Re: custom response writer which extends RawResponseWriter fails when shards > 1
Not that i can think of -- the existing API is really designed with the focus that the the ResponseWriter needs to specify the Content-Type prior to "writting" any bytes so Solr can do the best job possible "streaming" the data over the wire ... if you need to "pre-process" a lot of the response data before you know what you want to stream back, you have to do that in the getContentType() method .. but you can always store that pre-processed result in the SolrQueryRequest.getContext() and then write it out later. (you could in theory do a ton of processing to prepare a big byte[] of all the data you want to send out as part of getContentType(), and then your write() method could be a single line of code) : Date: Sun, 22 Apr 2018 15:49:30 +0100 : From: Lee Carroll <lee.a.carr...@googlemail.com> : Reply-To: solr-user@lucene.apache.org : To: solr-user@lucene.apache.org : Subject: Re: custom response writer which extends RawResponseWriter fails when : shards > 1 : : Hi, : I've ended up processing the doclist in the response twice. Once in the : write method and once in getContent. Its a bit inefficient but i'm only : looking at top doc each time so probably ok. : Is their a better way to do this ? : : Cheers lee C : : On 22 April 2018 at 13:26, Lee Carroll <lee.a.carr...@googlemail.com> wrote: : : > Hi, : > That works a treat. The raw response writer has a configurable base class : > which executes when no content stream is present in the response so I just : > delegate to that. I do still have an issue with writing content type on the : > http response from a value in the top document however. Although I do have : > a getContentType method which returns a value from a field in the top : > document this is called before the response writer has performed its write : > method. : > : > Whats the best way to set response headers using values from documents in : > the search result? In particular Content-Type. : > : > Cheers Lee C : > : > On 20 April 2018 at 20:03, Chris Hostetter <hossman_luc...@fucit.org> : > wrote: : > : >> : >> Invariant really means "invariant" ... nothing can change them : >> : >> In the case of "wt" this may seem weird and unhelpful, but the code that : >> handles defaults/appends/invariants is ignorant of what the params are. : >> : >> Since your writting custom code anyway, my suggestion would be that : >> perhaps you could make your custom ResponseWriter delegate to the javabin : >> responsewriter if/when you see that this is an "isShard=true" request? : >> : >> : >> : >> : Date: Thu, 19 Apr 2018 18:42:58 +0100 : >> : From: Lee Carroll <lee.a.carr...@googlemail.com> : >> : Reply-To: solr-user@lucene.apache.org : >> : To: solr-user@lucene.apache.org : >> : Subject: Re: custom response writer which extends RawResponseWriter : >> fails when : >> : shards > 1 : >> : : >> : Hi, : >> : : >> : I rewrote all of my tests to use SolrCloudTestCase rather than : >> SolrTestCaseJ4 : >> : and was able to replicate the responsewriter issue and debug with a : >> sharded : >> : collection. It turned out the issue was not with my response writer : >> really : >> : but rather my config. : >> : : >> : : >> : : >> : : >> : : >> : content : >> : : >> : : >> : : >> : : >> : In cloud mode having wt as an invariant breaks the collation of results : >> : from shards. Now I'm sure this is a common mistake which I've repeated : >> : (blush) but I do sort of want to actually implement my request handler : >> in : >> : this way. Is their a way to have a request handler support a single : >> : response writer but still work in cloud mode ? : >> : : >> : Could this be considered a bug ? : >> : : >> : Lee C : >> : : >> : On 18 April 2018 at 13:13, Mikhail Khludnev <m...@apache.org> wrote: : >> : : >> : > Injecting headers might require deeper customisation up to : >> establishing own : >> : > filter or so. : >> : > Speaking regarding your own WT, there might be some issues because : >> usually : >> : > it's not a big deal to use one wt for responding user query like : >> (wt=csv) : >> : > and wt=javabin in internal communication between aggregator and : >> slaves like : >> : > it happens in wt=csv query. : >> : > : >> : > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll < : >> lee.a.carr...@googlemail.com : >> : > > : >> : > wrote: : >> : > : >> : > > Inventive. I need to con
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Hi, I've ended up processing the doclist in the response twice. Once in the write method and once in getContent. Its a bit inefficient but i'm only looking at top doc each time so probably ok. Is their a better way to do this ? Cheers lee C On 22 April 2018 at 13:26, Lee Carroll <lee.a.carr...@googlemail.com> wrote: > Hi, > That works a treat. The raw response writer has a configurable base class > which executes when no content stream is present in the response so I just > delegate to that. I do still have an issue with writing content type on the > http response from a value in the top document however. Although I do have > a getContentType method which returns a value from a field in the top > document this is called before the response writer has performed its write > method. > > Whats the best way to set response headers using values from documents in > the search result? In particular Content-Type. > > Cheers Lee C > > On 20 April 2018 at 20:03, Chris Hostetter <hossman_luc...@fucit.org> > wrote: > >> >> Invariant really means "invariant" ... nothing can change them >> >> In the case of "wt" this may seem weird and unhelpful, but the code that >> handles defaults/appends/invariants is ignorant of what the params are. >> >> Since your writting custom code anyway, my suggestion would be that >> perhaps you could make your custom ResponseWriter delegate to the javabin >> responsewriter if/when you see that this is an "isShard=true" request? >> >> >> >> : Date: Thu, 19 Apr 2018 18:42:58 +0100 >> : From: Lee Carroll <lee.a.carr...@googlemail.com> >> : Reply-To: solr-user@lucene.apache.org >> : To: solr-user@lucene.apache.org >> : Subject: Re: custom response writer which extends RawResponseWriter >> fails when >> : shards > 1 >> : >> : Hi, >> : >> : I rewrote all of my tests to use SolrCloudTestCase rather than >> SolrTestCaseJ4 >> : and was able to replicate the responsewriter issue and debug with a >> sharded >> : collection. It turned out the issue was not with my response writer >> really >> : but rather my config. >> : >> : >> : >> : >> : >> : content >> : >> : >> : >> : >> : In cloud mode having wt as an invariant breaks the collation of results >> : from shards. Now I'm sure this is a common mistake which I've repeated >> : (blush) but I do sort of want to actually implement my request handler >> in >> : this way. Is their a way to have a request handler support a single >> : response writer but still work in cloud mode ? >> : >> : Could this be considered a bug ? >> : >> : Lee C >> : >> : On 18 April 2018 at 13:13, Mikhail Khludnev <m...@apache.org> wrote: >> : >> : > Injecting headers might require deeper customisation up to >> establishing own >> : > filter or so. >> : > Speaking regarding your own WT, there might be some issues because >> usually >> : > it's not a big deal to use one wt for responding user query like >> (wt=csv) >> : > and wt=javabin in internal communication between aggregator and >> slaves like >> : > it happens in wt=csv query. >> : > >> : > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll < >> lee.a.carr...@googlemail.com >> : > > >> : > wrote: >> : > >> : > > Inventive. I need to control content-type of the response from the >> : > document >> : > > field value. I have the actual content field and the content-type >> field >> : > to >> : > > use configured in the response writer. I've just noticed that the >> xslt >> : > > transformer allows you to do this but not controlled by document >> values. >> : > I >> : > > may also need to set some headers based on content-type and perhaps >> : > content >> : > > size, accept ranges comes to mind. Although I might be getting >> ahead of >> : > > myself. >> : > > >> : > > >> : > > >> : > > On 18 April 2018 at 12:05, Mikhail Khludnev <m...@apache.org> >> wrote: >> : > > >> : > > > well .. >> : > > > what if >> : > > > http://localhost:8983/solr/images/select?fl=content=id: >> : > > 1=1=csv& >> : > > > csv.separator==null >> : > > > ? >> : > > > >> : > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll < >> : >
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Hi, That works a treat. The raw response writer has a configurable base class which executes when no content stream is present in the response so I just delegate to that. I do still have an issue with writing content type on the http response from a value in the top document however. Although I do have a getContentType method which returns a value from a field in the top document this is called before the response writer has performed its write method. Whats the best way to set response headers using values from documents in the search result? In particular Content-Type. Cheers Lee C On 20 April 2018 at 20:03, Chris Hostetter <hossman_luc...@fucit.org> wrote: > > Invariant really means "invariant" ... nothing can change them > > In the case of "wt" this may seem weird and unhelpful, but the code that > handles defaults/appends/invariants is ignorant of what the params are. > > Since your writting custom code anyway, my suggestion would be that > perhaps you could make your custom ResponseWriter delegate to the javabin > responsewriter if/when you see that this is an "isShard=true" request? > > > > : Date: Thu, 19 Apr 2018 18:42:58 +0100 > : From: Lee Carroll <lee.a.carr...@googlemail.com> > : Reply-To: solr-user@lucene.apache.org > : To: solr-user@lucene.apache.org > : Subject: Re: custom response writer which extends RawResponseWriter > fails when > : shards > 1 > : > : Hi, > : > : I rewrote all of my tests to use SolrCloudTestCase rather than > SolrTestCaseJ4 > : and was able to replicate the responsewriter issue and debug with a > sharded > : collection. It turned out the issue was not with my response writer > really > : but rather my config. > : > : > : > : > : > : content > : > : > : > : > : In cloud mode having wt as an invariant breaks the collation of results > : from shards. Now I'm sure this is a common mistake which I've repeated > : (blush) but I do sort of want to actually implement my request handler in > : this way. Is their a way to have a request handler support a single > : response writer but still work in cloud mode ? > : > : Could this be considered a bug ? > : > : Lee C > : > : On 18 April 2018 at 13:13, Mikhail Khludnev <m...@apache.org> wrote: > : > : > Injecting headers might require deeper customisation up to > establishing own > : > filter or so. > : > Speaking regarding your own WT, there might be some issues because > usually > : > it's not a big deal to use one wt for responding user query like > (wt=csv) > : > and wt=javabin in internal communication between aggregator and slaves > like > : > it happens in wt=csv query. > : > > : > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll < > lee.a.carr...@googlemail.com > : > > > : > wrote: > : > > : > > Inventive. I need to control content-type of the response from the > : > document > : > > field value. I have the actual content field and the content-type > field > : > to > : > > use configured in the response writer. I've just noticed that the > xslt > : > > transformer allows you to do this but not controlled by document > values. > : > I > : > > may also need to set some headers based on content-type and perhaps > : > content > : > > size, accept ranges comes to mind. Although I might be getting ahead > of > : > > myself. > : > > > : > > > : > > > : > > On 18 April 2018 at 12:05, Mikhail Khludnev <m...@apache.org> wrote: > : > > > : > > > well .. > : > > > what if > : > > > http://localhost:8983/solr/images/select?fl=content=id: > : > > 1=1=csv& > : > > > csv.separator==null > : > > > ? > : > > > > : > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll < > : > > lee.a.carr...@googlemail.com > : > > > > > : > > > wrote: > : > > > > : > > > > sorry cut n paste error i'd get > : > > > > > : > > > > { > : > > > > "responseHeader":{ > : > > > > "zkConnected":true, > : > > > > "status":0, > : > > > > "QTime":0, > : > > > > "params":{ > : > > > > "q":"*:*", > : > > > > "fl":"content", > : > > > > "rows":"1"}}, > : > > > > "response":{"numFound":1,"start":0,"d
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Invariant really means "invariant" ... nothing can change them In the case of "wt" this may seem weird and unhelpful, but the code that handles defaults/appends/invariants is ignorant of what the params are. Since your writting custom code anyway, my suggestion would be that perhaps you could make your custom ResponseWriter delegate to the javabin responsewriter if/when you see that this is an "isShard=true" request? : Date: Thu, 19 Apr 2018 18:42:58 +0100 : From: Lee Carroll <lee.a.carr...@googlemail.com> : Reply-To: solr-user@lucene.apache.org : To: solr-user@lucene.apache.org : Subject: Re: custom response writer which extends RawResponseWriter fails when : shards > 1 : : Hi, : : I rewrote all of my tests to use SolrCloudTestCase rather than SolrTestCaseJ4 : and was able to replicate the responsewriter issue and debug with a sharded : collection. It turned out the issue was not with my response writer really : but rather my config. : : : : : : content : : : : : In cloud mode having wt as an invariant breaks the collation of results : from shards. Now I'm sure this is a common mistake which I've repeated : (blush) but I do sort of want to actually implement my request handler in : this way. Is their a way to have a request handler support a single : response writer but still work in cloud mode ? : : Could this be considered a bug ? : : Lee C : : On 18 April 2018 at 13:13, Mikhail Khludnev <m...@apache.org> wrote: : : > Injecting headers might require deeper customisation up to establishing own : > filter or so. : > Speaking regarding your own WT, there might be some issues because usually : > it's not a big deal to use one wt for responding user query like (wt=csv) : > and wt=javabin in internal communication between aggregator and slaves like : > it happens in wt=csv query. : > : > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll <lee.a.carr...@googlemail.com : > > : > wrote: : > : > > Inventive. I need to control content-type of the response from the : > document : > > field value. I have the actual content field and the content-type field : > to : > > use configured in the response writer. I've just noticed that the xslt : > > transformer allows you to do this but not controlled by document values. : > I : > > may also need to set some headers based on content-type and perhaps : > content : > > size, accept ranges comes to mind. Although I might be getting ahead of : > > myself. : > > : > > : > > : > > On 18 April 2018 at 12:05, Mikhail Khludnev <m...@apache.org> wrote: : > > : > > > well .. : > > > what if : > > > http://localhost:8983/solr/images/select?fl=content=id: : > > 1=1=csv& : > > > csv.separator==null : > > > ? : > > > : > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll < : > > lee.a.carr...@googlemail.com : > > > > : > > > wrote: : > > > : > > > > sorry cut n paste error i'd get : > > > > : > > > > { : > > > > "responseHeader":{ : > > > > "zkConnected":true, : > > > > "status":0, : > > > > "QTime":0, : > > > > "params":{ : > > > > "q":"*:*", : > > > > "fl":"content", : > > > > "rows":"1"}}, : > > > > "response":{"numFound":1,"start":0,"docs":[ : > > > > { : > > > > "content":"my-content-value"}] : > > > > }} : > > > > : > > > > : > > > > but you get my point : > > > > : > > > > : > > > > : > > > > On 18 April 2018 at 11:13, Lee Carroll <lee.a.carr...@googlemail.com : > > : > > > > wrote: : > > > > : > > > > > for http://localhost:8983/solr/images/select?fl=content=id: : > > 1=1 : > > > > > : > > > > > I'd get : > > > > > : > > > > > { : > > > > > "responseHeader":{ : > > > > > "zkConnected":true, : > > > > > "status":0, : > > > > > "QTime":1, : > > > > > "params":{ : > > > > > "q":"*:*", : > > > > > "_":"1524046333220"}}, : > > > > > "response":{"numFound":1,"start":0,"docs":[
Re: custom response writer which extends RawResponseWriter fails when shards > 1
You can introduce own searchHandler with wt invariant. When aggreator request slaves it will use regular /select with default wt=javabin (it's condrolled by shards.qt, btw) Providing such comprehensive application logic on top of neat solrconfig.xml is not the best idea, though. On Thu, Apr 19, 2018 at 9:57 PM, Lee Carrollwrote: > Default works. However in the case (which is maybe my case) where request > handler implies a response type and really should be locked down to prevent > abuse or error you could argue invariant is needed. > I guess its also not very elegant having an arbitrary rule, no wt as > invariant in cloud mode etc. > > > lee c > > On 19 April 2018 at 19:00, Mikhail Khludnev wrote: > > > what if you put it into "defaults"? > > > > On Thu, Apr 19, 2018 at 8:42 PM, Lee Carroll < > lee.a.carr...@googlemail.com > > > > > wrote: > > > > > Hi, > > > > > > I rewrote all of my tests to use SolrCloudTestCase rather than > > > SolrTestCaseJ4 > > > and was able to replicate the responsewriter issue and debug with a > > sharded > > > collection. It turned out the issue was not with my response writer > > really > > > but rather my config. > > > > > > > > > > > > > > > > > > content > > > > > > > > > > > > > > > In cloud mode having wt as an invariant breaks the collation of results > > > from shards. Now I'm sure this is a common mistake which I've repeated > > > (blush) but I do sort of want to actually implement my request handler > in > > > this way. Is their a way to have a request handler support a single > > > response writer but still work in cloud mode ? > > > > > > Could this be considered a bug ? > > > > > > Lee C > > > > > > On 18 April 2018 at 13:13, Mikhail Khludnev wrote: > > > > > > > Injecting headers might require deeper customisation up to > establishing > > > own > > > > filter or so. > > > > Speaking regarding your own WT, there might be some issues because > > > usually > > > > it's not a big deal to use one wt for responding user query like > > (wt=csv) > > > > and wt=javabin in internal communication between aggregator and > slaves > > > like > > > > it happens in wt=csv query. > > > > > > > > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll < > > > lee.a.carr...@googlemail.com > > > > > > > > > wrote: > > > > > > > > > Inventive. I need to control content-type of the response from the > > > > document > > > > > field value. I have the actual content field and the content-type > > field > > > > to > > > > > use configured in the response writer. I've just noticed that the > > xslt > > > > > transformer allows you to do this but not controlled by document > > > values. > > > > I > > > > > may also need to set some headers based on content-type and perhaps > > > > content > > > > > size, accept ranges comes to mind. Although I might be getting > ahead > > of > > > > > myself. > > > > > > > > > > > > > > > > > > > > On 18 April 2018 at 12:05, Mikhail Khludnev > wrote: > > > > > > > > > > > well .. > > > > > > what if > > > > > > http://localhost:8983/solr/images/select?fl=content=id: > > > > > 1=1=csv& > > > > > > csv.separator==null > > > > > > ? > > > > > > > > > > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll < > > > > > lee.a.carr...@googlemail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > sorry cut n paste error i'd get > > > > > > > > > > > > > > { > > > > > > > "responseHeader":{ > > > > > > > "zkConnected":true, > > > > > > > "status":0, > > > > > > > "QTime":0, > > > > > > > "params":{ > > > > > > > "q":"*:*", > > > > > > > "fl":"content", > > > > > > > "rows":"1"}}, > > > > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > > > > { > > > > > > > "content":"my-content-value"}] > > > > > > > }} > > > > > > > > > > > > > > > > > > > > > but you get my point > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 18 April 2018 at 11:13, Lee Carroll < > > > lee.a.carr...@googlemail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > for http://localhost:8983/solr/images/select?fl=content=id > : > > > > > 1=1 > > > > > > > > > > > > > > > > I'd get > > > > > > > > > > > > > > > > { > > > > > > > > "responseHeader":{ > > > > > > > > "zkConnected":true, > > > > > > > > "status":0, > > > > > > > > "QTime":1, > > > > > > > > "params":{ > > > > > > > > "q":"*:*", > > > > > > > > "_":"1524046333220"}}, > > > > > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > > > > > { > > > > > > > > "id":"1", > > > > > > > > "content":"my-content-value", > > > > > > > > "*content-type*":"text/plain"}] > > > > > > > > }} > > > > > > > > > > > > > > > > when i want > > > > > > > > > > > > > > > > my-content-value > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Default works. However in the case (which is maybe my case) where request handler implies a response type and really should be locked down to prevent abuse or error you could argue invariant is needed. I guess its also not very elegant having an arbitrary rule, no wt as invariant in cloud mode etc. lee c On 19 April 2018 at 19:00, Mikhail Khludnevwrote: > what if you put it into "defaults"? > > On Thu, Apr 19, 2018 at 8:42 PM, Lee Carroll > > wrote: > > > Hi, > > > > I rewrote all of my tests to use SolrCloudTestCase rather than > > SolrTestCaseJ4 > > and was able to replicate the responsewriter issue and debug with a > sharded > > collection. It turned out the issue was not with my response writer > really > > but rather my config. > > > > > > > > > > > > content > > > > > > > > > > In cloud mode having wt as an invariant breaks the collation of results > > from shards. Now I'm sure this is a common mistake which I've repeated > > (blush) but I do sort of want to actually implement my request handler in > > this way. Is their a way to have a request handler support a single > > response writer but still work in cloud mode ? > > > > Could this be considered a bug ? > > > > Lee C > > > > On 18 April 2018 at 13:13, Mikhail Khludnev wrote: > > > > > Injecting headers might require deeper customisation up to establishing > > own > > > filter or so. > > > Speaking regarding your own WT, there might be some issues because > > usually > > > it's not a big deal to use one wt for responding user query like > (wt=csv) > > > and wt=javabin in internal communication between aggregator and slaves > > like > > > it happens in wt=csv query. > > > > > > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll < > > lee.a.carr...@googlemail.com > > > > > > > wrote: > > > > > > > Inventive. I need to control content-type of the response from the > > > document > > > > field value. I have the actual content field and the content-type > field > > > to > > > > use configured in the response writer. I've just noticed that the > xslt > > > > transformer allows you to do this but not controlled by document > > values. > > > I > > > > may also need to set some headers based on content-type and perhaps > > > content > > > > size, accept ranges comes to mind. Although I might be getting ahead > of > > > > myself. > > > > > > > > > > > > > > > > On 18 April 2018 at 12:05, Mikhail Khludnev wrote: > > > > > > > > > well .. > > > > > what if > > > > > http://localhost:8983/solr/images/select?fl=content=id: > > > > 1=1=csv& > > > > > csv.separator==null > > > > > ? > > > > > > > > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll < > > > > lee.a.carr...@googlemail.com > > > > > > > > > > > wrote: > > > > > > > > > > > sorry cut n paste error i'd get > > > > > > > > > > > > { > > > > > > "responseHeader":{ > > > > > > "zkConnected":true, > > > > > > "status":0, > > > > > > "QTime":0, > > > > > > "params":{ > > > > > > "q":"*:*", > > > > > > "fl":"content", > > > > > > "rows":"1"}}, > > > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > > > { > > > > > > "content":"my-content-value"}] > > > > > > }} > > > > > > > > > > > > > > > > > > but you get my point > > > > > > > > > > > > > > > > > > > > > > > > On 18 April 2018 at 11:13, Lee Carroll < > > lee.a.carr...@googlemail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > for http://localhost:8983/solr/images/select?fl=content=id: > > > > 1=1 > > > > > > > > > > > > > > I'd get > > > > > > > > > > > > > > { > > > > > > > "responseHeader":{ > > > > > > > "zkConnected":true, > > > > > > > "status":0, > > > > > > > "QTime":1, > > > > > > > "params":{ > > > > > > > "q":"*:*", > > > > > > > "_":"1524046333220"}}, > > > > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > > > > { > > > > > > > "id":"1", > > > > > > > "content":"my-content-value", > > > > > > > "*content-type*":"text/plain"}] > > > > > > > }} > > > > > > > > > > > > > > when i want > > > > > > > > > > > > > > my-content-value > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev > > > wrote: > > > > > > > > > > > > > >> Lee, from this description I don see why it can't be addressed > > by > > > > > > fl,rows > > > > > > >> params. What makes it different form the typical Solr usage? > > > > > > >> > > > > > > >> > > > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll < > > > > > > >> lee.a.carr...@googlemail.com> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > Sure, we want to return a single field's value for the top > > > > matching > > > > > > >> > document for a given query. Bare content rather than a full > > > search > > > > > > >> result > > > > > > >> > listing. > > > > > > >> > > > > > > > >> > To be
Re: custom response writer which extends RawResponseWriter fails when shards > 1
what if you put it into "defaults"? On Thu, Apr 19, 2018 at 8:42 PM, Lee Carrollwrote: > Hi, > > I rewrote all of my tests to use SolrCloudTestCase rather than > SolrTestCaseJ4 > and was able to replicate the responsewriter issue and debug with a sharded > collection. It turned out the issue was not with my response writer really > but rather my config. > > > > > > content > > > > > In cloud mode having wt as an invariant breaks the collation of results > from shards. Now I'm sure this is a common mistake which I've repeated > (blush) but I do sort of want to actually implement my request handler in > this way. Is their a way to have a request handler support a single > response writer but still work in cloud mode ? > > Could this be considered a bug ? > > Lee C > > On 18 April 2018 at 13:13, Mikhail Khludnev wrote: > > > Injecting headers might require deeper customisation up to establishing > own > > filter or so. > > Speaking regarding your own WT, there might be some issues because > usually > > it's not a big deal to use one wt for responding user query like (wt=csv) > > and wt=javabin in internal communication between aggregator and slaves > like > > it happens in wt=csv query. > > > > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll < > lee.a.carr...@googlemail.com > > > > > wrote: > > > > > Inventive. I need to control content-type of the response from the > > document > > > field value. I have the actual content field and the content-type field > > to > > > use configured in the response writer. I've just noticed that the xslt > > > transformer allows you to do this but not controlled by document > values. > > I > > > may also need to set some headers based on content-type and perhaps > > content > > > size, accept ranges comes to mind. Although I might be getting ahead of > > > myself. > > > > > > > > > > > > On 18 April 2018 at 12:05, Mikhail Khludnev wrote: > > > > > > > well .. > > > > what if > > > > http://localhost:8983/solr/images/select?fl=content=id: > > > 1=1=csv& > > > > csv.separator==null > > > > ? > > > > > > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll < > > > lee.a.carr...@googlemail.com > > > > > > > > > wrote: > > > > > > > > > sorry cut n paste error i'd get > > > > > > > > > > { > > > > > "responseHeader":{ > > > > > "zkConnected":true, > > > > > "status":0, > > > > > "QTime":0, > > > > > "params":{ > > > > > "q":"*:*", > > > > > "fl":"content", > > > > > "rows":"1"}}, > > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > > { > > > > > "content":"my-content-value"}] > > > > > }} > > > > > > > > > > > > > > > but you get my point > > > > > > > > > > > > > > > > > > > > On 18 April 2018 at 11:13, Lee Carroll < > lee.a.carr...@googlemail.com > > > > > > > > wrote: > > > > > > > > > > > for http://localhost:8983/solr/images/select?fl=content=id: > > > 1=1 > > > > > > > > > > > > I'd get > > > > > > > > > > > > { > > > > > > "responseHeader":{ > > > > > > "zkConnected":true, > > > > > > "status":0, > > > > > > "QTime":1, > > > > > > "params":{ > > > > > > "q":"*:*", > > > > > > "_":"1524046333220"}}, > > > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > > > { > > > > > > "id":"1", > > > > > > "content":"my-content-value", > > > > > > "*content-type*":"text/plain"}] > > > > > > }} > > > > > > > > > > > > when i want > > > > > > > > > > > > my-content-value > > > > > > > > > > > > > > > > > > > > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev > > wrote: > > > > > > > > > > > >> Lee, from this description I don see why it can't be addressed > by > > > > > fl,rows > > > > > >> params. What makes it different form the typical Solr usage? > > > > > >> > > > > > >> > > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll < > > > > > >> lee.a.carr...@googlemail.com> > > > > > >> wrote: > > > > > >> > > > > > >> > Sure, we want to return a single field's value for the top > > > matching > > > > > >> > document for a given query. Bare content rather than a full > > search > > > > > >> result > > > > > >> > listing. > > > > > >> > > > > > > >> > To be concrete: > > > > > >> > > > > > > >> > For a schema of fields id [unique key], > > > > content[stored],content-type[ > > > > > >> > stored] > > > > > >> > For a request: > > > > > >> > > > > > > >> >1. Request URL: > > > > > >> >https://localhost/solr/content?q=id:1 > > > > > >> >2. Request Method: > > > > > >> >GET > > > > > >> > > > > > > >> > We get a response > > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: > > [content-type > > > > > value] > > > > > >> > > > > > > >> > and the body to be the raw value of content > > > > > >> > > > > > > >> > In short clients consume directly the most relevant "content" > > > > returned > > > > > >> from > > > > > >> >
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Hi, I rewrote all of my tests to use SolrCloudTestCase rather than SolrTestCaseJ4 and was able to replicate the responsewriter issue and debug with a sharded collection. It turned out the issue was not with my response writer really but rather my config. content In cloud mode having wt as an invariant breaks the collation of results from shards. Now I'm sure this is a common mistake which I've repeated (blush) but I do sort of want to actually implement my request handler in this way. Is their a way to have a request handler support a single response writer but still work in cloud mode ? Could this be considered a bug ? Lee C On 18 April 2018 at 13:13, Mikhail Khludnevwrote: > Injecting headers might require deeper customisation up to establishing own > filter or so. > Speaking regarding your own WT, there might be some issues because usually > it's not a big deal to use one wt for responding user query like (wt=csv) > and wt=javabin in internal communication between aggregator and slaves like > it happens in wt=csv query. > > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll > > wrote: > > > Inventive. I need to control content-type of the response from the > document > > field value. I have the actual content field and the content-type field > to > > use configured in the response writer. I've just noticed that the xslt > > transformer allows you to do this but not controlled by document values. > I > > may also need to set some headers based on content-type and perhaps > content > > size, accept ranges comes to mind. Although I might be getting ahead of > > myself. > > > > > > > > On 18 April 2018 at 12:05, Mikhail Khludnev wrote: > > > > > well .. > > > what if > > > http://localhost:8983/solr/images/select?fl=content=id: > > 1=1=csv& > > > csv.separator==null > > > ? > > > > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll < > > lee.a.carr...@googlemail.com > > > > > > > wrote: > > > > > > > sorry cut n paste error i'd get > > > > > > > > { > > > > "responseHeader":{ > > > > "zkConnected":true, > > > > "status":0, > > > > "QTime":0, > > > > "params":{ > > > > "q":"*:*", > > > > "fl":"content", > > > > "rows":"1"}}, > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > { > > > > "content":"my-content-value"}] > > > > }} > > > > > > > > > > > > but you get my point > > > > > > > > > > > > > > > > On 18 April 2018 at 11:13, Lee Carroll > > > > > wrote: > > > > > > > > > for http://localhost:8983/solr/images/select?fl=content=id: > > 1=1 > > > > > > > > > > I'd get > > > > > > > > > > { > > > > > "responseHeader":{ > > > > > "zkConnected":true, > > > > > "status":0, > > > > > "QTime":1, > > > > > "params":{ > > > > > "q":"*:*", > > > > > "_":"1524046333220"}}, > > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > > { > > > > > "id":"1", > > > > > "content":"my-content-value", > > > > > "*content-type*":"text/plain"}] > > > > > }} > > > > > > > > > > when i want > > > > > > > > > > my-content-value > > > > > > > > > > > > > > > > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev > wrote: > > > > > > > > > >> Lee, from this description I don see why it can't be addressed by > > > > fl,rows > > > > >> params. What makes it different form the typical Solr usage? > > > > >> > > > > >> > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll < > > > > >> lee.a.carr...@googlemail.com> > > > > >> wrote: > > > > >> > > > > >> > Sure, we want to return a single field's value for the top > > matching > > > > >> > document for a given query. Bare content rather than a full > search > > > > >> result > > > > >> > listing. > > > > >> > > > > > >> > To be concrete: > > > > >> > > > > > >> > For a schema of fields id [unique key], > > > content[stored],content-type[ > > > > >> > stored] > > > > >> > For a request: > > > > >> > > > > > >> >1. Request URL: > > > > >> >https://localhost/solr/content?q=id:1 > > > > >> >2. Request Method: > > > > >> >GET > > > > >> > > > > > >> > We get a response > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: > [content-type > > > > value] > > > > >> > > > > > >> > and the body to be the raw value of content > > > > >> > > > > > >> > In short clients consume directly the most relevant "content" > > > returned > > > > >> from > > > > >> > solr queries they construct. > > > > >> > > > > > >> > Naively I've implemented a subclass of RawResponseWriter which > > takes > > > > the > > > > >> > first docs values and adds them to the appended "content" > stream. > > > > >> Should I > > > > >> > selectively add the content stream depending on if this is the > > final > > > > >> > aggregation of cloud results (and provide a base class writer to > > act > > > > if > > > > >> > not), if so
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Injecting headers might require deeper customisation up to establishing own filter or so. Speaking regarding your own WT, there might be some issues because usually it's not a big deal to use one wt for responding user query like (wt=csv) and wt=javabin in internal communication between aggregator and slaves like it happens in wt=csv query. On Wed, Apr 18, 2018 at 2:19 PM, Lee Carrollwrote: > Inventive. I need to control content-type of the response from the document > field value. I have the actual content field and the content-type field to > use configured in the response writer. I've just noticed that the xslt > transformer allows you to do this but not controlled by document values. I > may also need to set some headers based on content-type and perhaps content > size, accept ranges comes to mind. Although I might be getting ahead of > myself. > > > > On 18 April 2018 at 12:05, Mikhail Khludnev wrote: > > > well .. > > what if > > http://localhost:8983/solr/images/select?fl=content=id: > 1=1=csv& > > csv.separator==null > > ? > > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll < > lee.a.carr...@googlemail.com > > > > > wrote: > > > > > sorry cut n paste error i'd get > > > > > > { > > > "responseHeader":{ > > > "zkConnected":true, > > > "status":0, > > > "QTime":0, > > > "params":{ > > > "q":"*:*", > > > "fl":"content", > > > "rows":"1"}}, > > > "response":{"numFound":1,"start":0,"docs":[ > > > { > > > "content":"my-content-value"}] > > > }} > > > > > > > > > but you get my point > > > > > > > > > > > > On 18 April 2018 at 11:13, Lee Carroll > > > wrote: > > > > > > > for http://localhost:8983/solr/images/select?fl=content=id: > 1=1 > > > > > > > > I'd get > > > > > > > > { > > > > "responseHeader":{ > > > > "zkConnected":true, > > > > "status":0, > > > > "QTime":1, > > > > "params":{ > > > > "q":"*:*", > > > > "_":"1524046333220"}}, > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > { > > > > "id":"1", > > > > "content":"my-content-value", > > > > "*content-type*":"text/plain"}] > > > > }} > > > > > > > > when i want > > > > > > > > my-content-value > > > > > > > > > > > > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev wrote: > > > > > > > >> Lee, from this description I don see why it can't be addressed by > > > fl,rows > > > >> params. What makes it different form the typical Solr usage? > > > >> > > > >> > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll < > > > >> lee.a.carr...@googlemail.com> > > > >> wrote: > > > >> > > > >> > Sure, we want to return a single field's value for the top > matching > > > >> > document for a given query. Bare content rather than a full search > > > >> result > > > >> > listing. > > > >> > > > > >> > To be concrete: > > > >> > > > > >> > For a schema of fields id [unique key], > > content[stored],content-type[ > > > >> > stored] > > > >> > For a request: > > > >> > > > > >> >1. Request URL: > > > >> >https://localhost/solr/content?q=id:1 > > > >> >2. Request Method: > > > >> >GET > > > >> > > > > >> > We get a response > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type > > > value] > > > >> > > > > >> > and the body to be the raw value of content > > > >> > > > > >> > In short clients consume directly the most relevant "content" > > returned > > > >> from > > > >> > solr queries they construct. > > > >> > > > > >> > Naively I've implemented a subclass of RawResponseWriter which > takes > > > the > > > >> > first docs values and adds them to the appended "content" stream. > > > >> Should I > > > >> > selectively add the content stream depending on if this is the > final > > > >> > aggregation of cloud results (and provide a base class writer to > act > > > if > > > >> > not), if so how do I know its the final aggregation. Or is adding > > the > > > >> > content stream within the response writer a bad idea. Should that > be > > > >> being > > > >> > added to the response somewhere else? > > > >> > > > > >> > Failing all of the above is asking about response writer an X / Y > > > >> problem. > > > >> > Is their a better way to achieve the above. I'd looked at > > transforming > > > >> > response xml but that seemed not to offer a complete bare slate. > > > >> > > > > >> > Cheers Lee C > > > >> > > > > >> > > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev > > wrote: > > > >> > > > > >> > > In distributed search response writer is used twice > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed- > > requests.html > > > >> > > once slave node that's where response writer yields "json" > content > > > >> and it > > > >> > > upset aggregator node which is expect only javabin. > > > >> > > I hardly can comment on rrw, it's probably used for responding > > > >>
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Inventive. I need to control content-type of the response from the document field value. I have the actual content field and the content-type field to use configured in the response writer. I've just noticed that the xslt transformer allows you to do this but not controlled by document values. I may also need to set some headers based on content-type and perhaps content size, accept ranges comes to mind. Although I might be getting ahead of myself. On 18 April 2018 at 12:05, Mikhail Khludnevwrote: > well .. > what if > http://localhost:8983/solr/images/select?fl=content=id:1=1=csv; > csv.separator==null > ? > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll > > wrote: > > > sorry cut n paste error i'd get > > > > { > > "responseHeader":{ > > "zkConnected":true, > > "status":0, > > "QTime":0, > > "params":{ > > "q":"*:*", > > "fl":"content", > > "rows":"1"}}, > > "response":{"numFound":1,"start":0,"docs":[ > > { > > "content":"my-content-value"}] > > }} > > > > > > but you get my point > > > > > > > > On 18 April 2018 at 11:13, Lee Carroll > > wrote: > > > > > for http://localhost:8983/solr/images/select?fl=content=id:1=1 > > > > > > I'd get > > > > > > { > > > "responseHeader":{ > > > "zkConnected":true, > > > "status":0, > > > "QTime":1, > > > "params":{ > > > "q":"*:*", > > > "_":"1524046333220"}}, > > > "response":{"numFound":1,"start":0,"docs":[ > > > { > > > "id":"1", > > > "content":"my-content-value", > > > "*content-type*":"text/plain"}] > > > }} > > > > > > when i want > > > > > > my-content-value > > > > > > > > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev wrote: > > > > > >> Lee, from this description I don see why it can't be addressed by > > fl,rows > > >> params. What makes it different form the typical Solr usage? > > >> > > >> > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll < > > >> lee.a.carr...@googlemail.com> > > >> wrote: > > >> > > >> > Sure, we want to return a single field's value for the top matching > > >> > document for a given query. Bare content rather than a full search > > >> result > > >> > listing. > > >> > > > >> > To be concrete: > > >> > > > >> > For a schema of fields id [unique key], > content[stored],content-type[ > > >> > stored] > > >> > For a request: > > >> > > > >> >1. Request URL: > > >> >https://localhost/solr/content?q=id:1 > > >> >2. Request Method: > > >> >GET > > >> > > > >> > We get a response > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type > > value] > > >> > > > >> > and the body to be the raw value of content > > >> > > > >> > In short clients consume directly the most relevant "content" > returned > > >> from > > >> > solr queries they construct. > > >> > > > >> > Naively I've implemented a subclass of RawResponseWriter which takes > > the > > >> > first docs values and adds them to the appended "content" stream. > > >> Should I > > >> > selectively add the content stream depending on if this is the final > > >> > aggregation of cloud results (and provide a base class writer to act > > if > > >> > not), if so how do I know its the final aggregation. Or is adding > the > > >> > content stream within the response writer a bad idea. Should that be > > >> being > > >> > added to the response somewhere else? > > >> > > > >> > Failing all of the above is asking about response writer an X / Y > > >> problem. > > >> > Is their a better way to achieve the above. I'd looked at > transforming > > >> > response xml but that seemed not to offer a complete bare slate. > > >> > > > >> > Cheers Lee C > > >> > > > >> > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev > wrote: > > >> > > > >> > > In distributed search response writer is used twice > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed- > requests.html > > >> > > once slave node that's where response writer yields "json" content > > >> and it > > >> > > upset aggregator node which is expect only javabin. > > >> > > I hardly can comment on rrw, it's probably used for responding > > >> separate > > >> > > files in distrib=false mode. > > >> > > You can start from describing why you need to create own response > > >> writer. > > >> > > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll < > > >> > lee.a.carr...@googlemail.com > > >> > > > > > >> > > wrote: > > >> > > > > >> > > > Ok. My expectation was the response writer would not be used > until > > >> the > > >> > > > final serialization of the result. If my response writer breaks > > the > > >> > > > response writer contract, exactly the way rawResponseWriter does > > and > > >> > just > > >> > > > out puts a filed value how does that work? Does > rawResponseWriter > > >> > support > > >> > > > cloud mode? > > >> > > > > > >> > > > > > >> > > > > >
Re: custom response writer which extends RawResponseWriter fails when shards > 1
well .. what if http://localhost:8983/solr/images/select?fl=content=id:1=1=csv; csv.separator==null ? On Wed, Apr 18, 2018 at 1:18 PM, Lee Carrollwrote: > sorry cut n paste error i'd get > > { > "responseHeader":{ > "zkConnected":true, > "status":0, > "QTime":0, > "params":{ > "q":"*:*", > "fl":"content", > "rows":"1"}}, > "response":{"numFound":1,"start":0,"docs":[ > { > "content":"my-content-value"}] > }} > > > but you get my point > > > > On 18 April 2018 at 11:13, Lee Carroll > wrote: > > > for http://localhost:8983/solr/images/select?fl=content=id:1=1 > > > > I'd get > > > > { > > "responseHeader":{ > > "zkConnected":true, > > "status":0, > > "QTime":1, > > "params":{ > > "q":"*:*", > > "_":"1524046333220"}}, > > "response":{"numFound":1,"start":0,"docs":[ > > { > > "id":"1", > > "content":"my-content-value", > > "*content-type*":"text/plain"}] > > }} > > > > when i want > > > > my-content-value > > > > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev wrote: > > > >> Lee, from this description I don see why it can't be addressed by > fl,rows > >> params. What makes it different form the typical Solr usage? > >> > >> > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll < > >> lee.a.carr...@googlemail.com> > >> wrote: > >> > >> > Sure, we want to return a single field's value for the top matching > >> > document for a given query. Bare content rather than a full search > >> result > >> > listing. > >> > > >> > To be concrete: > >> > > >> > For a schema of fields id [unique key], content[stored],content-type[ > >> > stored] > >> > For a request: > >> > > >> >1. Request URL: > >> >https://localhost/solr/content?q=id:1 > >> >2. Request Method: > >> >GET > >> > > >> > We get a response > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type > value] > >> > > >> > and the body to be the raw value of content > >> > > >> > In short clients consume directly the most relevant "content" returned > >> from > >> > solr queries they construct. > >> > > >> > Naively I've implemented a subclass of RawResponseWriter which takes > the > >> > first docs values and adds them to the appended "content" stream. > >> Should I > >> > selectively add the content stream depending on if this is the final > >> > aggregation of cloud results (and provide a base class writer to act > if > >> > not), if so how do I know its the final aggregation. Or is adding the > >> > content stream within the response writer a bad idea. Should that be > >> being > >> > added to the response somewhere else? > >> > > >> > Failing all of the above is asking about response writer an X / Y > >> problem. > >> > Is their a better way to achieve the above. I'd looked at transforming > >> > response xml but that seemed not to offer a complete bare slate. > >> > > >> > Cheers Lee C > >> > > >> > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev wrote: > >> > > >> > > In distributed search response writer is used twice > >> > > https://lucene.apache.org/solr/guide/7_1/distributed-requests.html > >> > > once slave node that's where response writer yields "json" content > >> and it > >> > > upset aggregator node which is expect only javabin. > >> > > I hardly can comment on rrw, it's probably used for responding > >> separate > >> > > files in distrib=false mode. > >> > > You can start from describing why you need to create own response > >> writer. > >> > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll < > >> > lee.a.carr...@googlemail.com > >> > > > > >> > > wrote: > >> > > > >> > > > Ok. My expectation was the response writer would not be used until > >> the > >> > > > final serialization of the result. If my response writer breaks > the > >> > > > response writer contract, exactly the way rawResponseWriter does > and > >> > just > >> > > > out puts a filed value how does that work? Does rawResponseWriter > >> > support > >> > > > cloud mode? > >> > > > > >> > > > > >> > > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev > >> wrote: > >> > > > > >> > > > > That's what should happen. > >> > > > > > >> > > > > Expected mime type application/octet-stream but got > >> application/json. > >> > > > > > >> > > > > Distributed search coordinator expect to merge slave responses > in > >> > > javabin > >> > > > > format. But slave's wt indicated json. > >> > > > > As far as I know only javabin might be used to distributed > search > >> > > > > underneath. Coordinator itself might yield json. > >> > > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll < > >> > > > lee.a.carr...@googlemail.com > >> > > > > > > >> > > > > wrote: > >> > > > > > >> > > > > > Sure > >> > > > > > > >> > > > > > with 1 shard 1 replica this request works fine > >> > > > > > > >> > > > > >1.
Re: custom response writer which extends RawResponseWriter fails when shards > 1
sorry cut n paste error i'd get { "responseHeader":{ "zkConnected":true, "status":0, "QTime":0, "params":{ "q":"*:*", "fl":"content", "rows":"1"}}, "response":{"numFound":1,"start":0,"docs":[ { "content":"my-content-value"}] }} but you get my point On 18 April 2018 at 11:13, Lee Carrollwrote: > for http://localhost:8983/solr/images/select?fl=content=id:1=1 > > I'd get > > { > "responseHeader":{ > "zkConnected":true, > "status":0, > "QTime":1, > "params":{ > "q":"*:*", > "_":"1524046333220"}}, > "response":{"numFound":1,"start":0,"docs":[ > { > "id":"1", > "content":"my-content-value", > "*content-type*":"text/plain"}] > }} > > when i want > > my-content-value > > > > On 18 April 2018 at 10:55, Mikhail Khludnev wrote: > >> Lee, from this description I don see why it can't be addressed by fl,rows >> params. What makes it different form the typical Solr usage? >> >> >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll < >> lee.a.carr...@googlemail.com> >> wrote: >> >> > Sure, we want to return a single field's value for the top matching >> > document for a given query. Bare content rather than a full search >> result >> > listing. >> > >> > To be concrete: >> > >> > For a schema of fields id [unique key], content[stored],content-type[ >> > stored] >> > For a request: >> > >> >1. Request URL: >> >https://localhost/solr/content?q=id:1 >> >2. Request Method: >> >GET >> > >> > We get a response >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type value] >> > >> > and the body to be the raw value of content >> > >> > In short clients consume directly the most relevant "content" returned >> from >> > solr queries they construct. >> > >> > Naively I've implemented a subclass of RawResponseWriter which takes the >> > first docs values and adds them to the appended "content" stream. >> Should I >> > selectively add the content stream depending on if this is the final >> > aggregation of cloud results (and provide a base class writer to act if >> > not), if so how do I know its the final aggregation. Or is adding the >> > content stream within the response writer a bad idea. Should that be >> being >> > added to the response somewhere else? >> > >> > Failing all of the above is asking about response writer an X / Y >> problem. >> > Is their a better way to achieve the above. I'd looked at transforming >> > response xml but that seemed not to offer a complete bare slate. >> > >> > Cheers Lee C >> > >> > >> > On 17 April 2018 at 21:36, Mikhail Khludnev wrote: >> > >> > > In distributed search response writer is used twice >> > > https://lucene.apache.org/solr/guide/7_1/distributed-requests.html >> > > once slave node that's where response writer yields "json" content >> and it >> > > upset aggregator node which is expect only javabin. >> > > I hardly can comment on rrw, it's probably used for responding >> separate >> > > files in distrib=false mode. >> > > You can start from describing why you need to create own response >> writer. >> > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll < >> > lee.a.carr...@googlemail.com >> > > > >> > > wrote: >> > > >> > > > Ok. My expectation was the response writer would not be used until >> the >> > > > final serialization of the result. If my response writer breaks the >> > > > response writer contract, exactly the way rawResponseWriter does and >> > just >> > > > out puts a filed value how does that work? Does rawResponseWriter >> > support >> > > > cloud mode? >> > > > >> > > > >> > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev >> wrote: >> > > > >> > > > > That's what should happen. >> > > > > >> > > > > Expected mime type application/octet-stream but got >> application/json. >> > > > > >> > > > > Distributed search coordinator expect to merge slave responses in >> > > javabin >> > > > > format. But slave's wt indicated json. >> > > > > As far as I know only javabin might be used to distributed search >> > > > > underneath. Coordinator itself might yield json. >> > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll < >> > > > lee.a.carr...@googlemail.com >> > > > > > >> > > > > wrote: >> > > > > >> > > > > > Sure >> > > > > > >> > > > > > with 1 shard 1 replica this request works fine >> > > > > > >> > > > > >1. Request URL: >> > > > > >http://localhost:8983/solr/images/image?q=id:1 >> > > > > >2. Request Method: >> > > > > >GET >> > > > > >3. Status Code: >> > > > > >200 OK >> > > > > > >> > > > > > logs are clean >> > > > > > >> > > > > > with 2 shards 2 replicas the same request fails and in the logs >> > > > > > >> > > > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7 >> > > > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore; >> > > > > >
Re: custom response writer which extends RawResponseWriter fails when shards > 1
for http://localhost:8983/solr/images/select?fl=content=id:1=1 I'd get { "responseHeader":{ "zkConnected":true, "status":0, "QTime":1, "params":{ "q":"*:*", "_":"1524046333220"}}, "response":{"numFound":1,"start":0,"docs":[ { "id":"1", "content":"my-content-value", "*content-type*":"text/plain"}] }} when i want my-content-value On 18 April 2018 at 10:55, Mikhail Khludnevwrote: > Lee, from this description I don see why it can't be addressed by fl,rows > params. What makes it different form the typical Solr usage? > > > On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll < > lee.a.carr...@googlemail.com> > wrote: > > > Sure, we want to return a single field's value for the top matching > > document for a given query. Bare content rather than a full search result > > listing. > > > > To be concrete: > > > > For a schema of fields id [unique key], content[stored],content-type[ > > stored] > > For a request: > > > >1. Request URL: > >https://localhost/solr/content?q=id:1 > >2. Request Method: > >GET > > > > We get a response > > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type value] > > > > and the body to be the raw value of content > > > > In short clients consume directly the most relevant "content" returned > from > > solr queries they construct. > > > > Naively I've implemented a subclass of RawResponseWriter which takes the > > first docs values and adds them to the appended "content" stream. Should > I > > selectively add the content stream depending on if this is the final > > aggregation of cloud results (and provide a base class writer to act if > > not), if so how do I know its the final aggregation. Or is adding the > > content stream within the response writer a bad idea. Should that be > being > > added to the response somewhere else? > > > > Failing all of the above is asking about response writer an X / Y > problem. > > Is their a better way to achieve the above. I'd looked at transforming > > response xml but that seemed not to offer a complete bare slate. > > > > Cheers Lee C > > > > > > On 17 April 2018 at 21:36, Mikhail Khludnev wrote: > > > > > In distributed search response writer is used twice > > > https://lucene.apache.org/solr/guide/7_1/distributed-requests.html > > > once slave node that's where response writer yields "json" content and > it > > > upset aggregator node which is expect only javabin. > > > I hardly can comment on rrw, it's probably used for responding separate > > > files in distrib=false mode. > > > You can start from describing why you need to create own response > writer. > > > > > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll < > > lee.a.carr...@googlemail.com > > > > > > > wrote: > > > > > > > Ok. My expectation was the response writer would not be used until > the > > > > final serialization of the result. If my response writer breaks the > > > > response writer contract, exactly the way rawResponseWriter does and > > just > > > > out puts a filed value how does that work? Does rawResponseWriter > > support > > > > cloud mode? > > > > > > > > > > > > > > > > On 17 April 2018 at 15:55, Mikhail Khludnev wrote: > > > > > > > > > That's what should happen. > > > > > > > > > > Expected mime type application/octet-stream but got > application/json. > > > > > > > > > > Distributed search coordinator expect to merge slave responses in > > > javabin > > > > > format. But slave's wt indicated json. > > > > > As far as I know only javabin might be used to distributed search > > > > > underneath. Coordinator itself might yield json. > > > > > > > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll < > > > > lee.a.carr...@googlemail.com > > > > > > > > > > > wrote: > > > > > > > > > > > Sure > > > > > > > > > > > > with 1 shard 1 replica this request works fine > > > > > > > > > > > >1. Request URL: > > > > > >http://localhost:8983/solr/images/image?q=id:1 > > > > > >2. Request Method: > > > > > >GET > > > > > >3. Status Code: > > > > > >200 OK > > > > > > > > > > > > logs are clean > > > > > > > > > > > > with 2 shards 2 replicas the same request fails and in the logs > > > > > > > > > > > > > > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7 > > > > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore; > > > > > > [images_shard2_replica_n4] webapp=/solr path=/image > > > > > > params={df=text=false=/image=id=score& > > > > > > shards.purpose=4=0=true= > > > > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/ > > > > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/ > > > > > > =10=2=id:1=1523971232039=true= > > javabin} > > > > > > hits=0 status=0 QTime=0 > > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 > > > > > > x:images_shard1_replica_n1] org.apache.solr.common. > SolrException; > > > > > >
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Lee, from this description I don see why it can't be addressed by fl,rows params. What makes it different form the typical Solr usage? On Wed, Apr 18, 2018 at 12:31 PM, Lee Carrollwrote: > Sure, we want to return a single field's value for the top matching > document for a given query. Bare content rather than a full search result > listing. > > To be concrete: > > For a schema of fields id [unique key], content[stored],content-type[ > stored] > For a request: > >1. Request URL: >https://localhost/solr/content?q=id:1 >2. Request Method: >GET > > We get a response > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type value] > > and the body to be the raw value of content > > In short clients consume directly the most relevant "content" returned from > solr queries they construct. > > Naively I've implemented a subclass of RawResponseWriter which takes the > first docs values and adds them to the appended "content" stream. Should I > selectively add the content stream depending on if this is the final > aggregation of cloud results (and provide a base class writer to act if > not), if so how do I know its the final aggregation. Or is adding the > content stream within the response writer a bad idea. Should that be being > added to the response somewhere else? > > Failing all of the above is asking about response writer an X / Y problem. > Is their a better way to achieve the above. I'd looked at transforming > response xml but that seemed not to offer a complete bare slate. > > Cheers Lee C > > > On 17 April 2018 at 21:36, Mikhail Khludnev wrote: > > > In distributed search response writer is used twice > > https://lucene.apache.org/solr/guide/7_1/distributed-requests.html > > once slave node that's where response writer yields "json" content and it > > upset aggregator node which is expect only javabin. > > I hardly can comment on rrw, it's probably used for responding separate > > files in distrib=false mode. > > You can start from describing why you need to create own response writer. > > > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll < > lee.a.carr...@googlemail.com > > > > > wrote: > > > > > Ok. My expectation was the response writer would not be used until the > > > final serialization of the result. If my response writer breaks the > > > response writer contract, exactly the way rawResponseWriter does and > just > > > out puts a filed value how does that work? Does rawResponseWriter > support > > > cloud mode? > > > > > > > > > > > > On 17 April 2018 at 15:55, Mikhail Khludnev wrote: > > > > > > > That's what should happen. > > > > > > > > Expected mime type application/octet-stream but got application/json. > > > > > > > > Distributed search coordinator expect to merge slave responses in > > javabin > > > > format. But slave's wt indicated json. > > > > As far as I know only javabin might be used to distributed search > > > > underneath. Coordinator itself might yield json. > > > > > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll < > > > lee.a.carr...@googlemail.com > > > > > > > > > wrote: > > > > > > > > > Sure > > > > > > > > > > with 1 shard 1 replica this request works fine > > > > > > > > > >1. Request URL: > > > > >http://localhost:8983/solr/images/image?q=id:1 > > > > >2. Request Method: > > > > >GET > > > > >3. Status Code: > > > > >200 OK > > > > > > > > > > logs are clean > > > > > > > > > > with 2 shards 2 replicas the same request fails and in the logs > > > > > > > > > > > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7 > > > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore; > > > > > [images_shard2_replica_n4] webapp=/solr path=/image > > > > > params={df=text=false=/image=id=score& > > > > > shards.purpose=4=0=true= > > > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/ > > > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/ > > > > > =10=2=id:1=1523971232039=true= > javabin} > > > > > hits=0 status=0 QTime=0 > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 > > > > > x:images_shard1_replica_n1] org.apache.solr.common.SolrException; > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$ > > RemoteSolrException: > > > > > Error > > > > > from server at http://10.224.30.207:8983/ > > solr/images_shard2_replica_n4 > > > : > > > > > Expected mime type application/octet-stream but got > application/json. > > > > > at > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient. > > > > > executeMethod(HttpSolrClient.java:607) > > > > > at > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > > > HttpSolrClient.java:255) > > > > > at > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > > > HttpSolrClient.java:244) > > > > > at > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > > >
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Sure, we want to return a single field's value for the top matching document for a given query. Bare content rather than a full search result listing. To be concrete: For a schema of fields id [unique key], content[stored],content-type[stored] For a request: 1. Request URL: https://localhost/solr/content?q=id:1 2. Request Method: GET We get a response HTTP/1.1 200 OK Content-Length: 16261 Content-Type: [content-type value] and the body to be the raw value of content In short clients consume directly the most relevant "content" returned from solr queries they construct. Naively I've implemented a subclass of RawResponseWriter which takes the first docs values and adds them to the appended "content" stream. Should I selectively add the content stream depending on if this is the final aggregation of cloud results (and provide a base class writer to act if not), if so how do I know its the final aggregation. Or is adding the content stream within the response writer a bad idea. Should that be being added to the response somewhere else? Failing all of the above is asking about response writer an X / Y problem. Is their a better way to achieve the above. I'd looked at transforming response xml but that seemed not to offer a complete bare slate. Cheers Lee C On 17 April 2018 at 21:36, Mikhail Khludnevwrote: > In distributed search response writer is used twice > https://lucene.apache.org/solr/guide/7_1/distributed-requests.html > once slave node that's where response writer yields "json" content and it > upset aggregator node which is expect only javabin. > I hardly can comment on rrw, it's probably used for responding separate > files in distrib=false mode. > You can start from describing why you need to create own response writer. > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll > > wrote: > > > Ok. My expectation was the response writer would not be used until the > > final serialization of the result. If my response writer breaks the > > response writer contract, exactly the way rawResponseWriter does and just > > out puts a filed value how does that work? Does rawResponseWriter support > > cloud mode? > > > > > > > > On 17 April 2018 at 15:55, Mikhail Khludnev wrote: > > > > > That's what should happen. > > > > > > Expected mime type application/octet-stream but got application/json. > > > > > > Distributed search coordinator expect to merge slave responses in > javabin > > > format. But slave's wt indicated json. > > > As far as I know only javabin might be used to distributed search > > > underneath. Coordinator itself might yield json. > > > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll < > > lee.a.carr...@googlemail.com > > > > > > > wrote: > > > > > > > Sure > > > > > > > > with 1 shard 1 replica this request works fine > > > > > > > >1. Request URL: > > > >http://localhost:8983/solr/images/image?q=id:1 > > > >2. Request Method: > > > >GET > > > >3. Status Code: > > > >200 OK > > > > > > > > logs are clean > > > > > > > > with 2 shards 2 replicas the same request fails and in the logs > > > > > > > > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7 > > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore; > > > > [images_shard2_replica_n4] webapp=/solr path=/image > > > > params={df=text=false=/image=id=score& > > > > shards.purpose=4=0=true= > > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/ > > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/ > > > > =10=2=id:1=1523971232039=true=javabin} > > > > hits=0 status=0 QTime=0 > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 > > > > x:images_shard1_replica_n1] org.apache.solr.common.SolrException; > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$ > RemoteSolrException: > > > > Error > > > > from server at http://10.224.30.207:8983/ > solr/images_shard2_replica_n4 > > : > > > > Expected mime type application/octet-stream but got application/json. > > > > at > > > > org.apache.solr.client.solrj.impl.HttpSolrClient. > > > > executeMethod(HttpSolrClient.java:607) > > > > at > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > > HttpSolrClient.java:255) > > > > at > > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > > HttpSolrClient.java:244) > > > > at > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > > doRequest(LBHttpSolrClient.java:483) > > > > at > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request( > > > > LBHttpSolrClient.java:413) > > > > at > > > > org.apache.solr.handler.component.HttpShardHandlerFactory. > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273) > > > > at > > > > org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0( > > > > HttpShardHandler.java:175) > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > > at
Re: custom response writer which extends RawResponseWriter fails when shards > 1
In distributed search response writer is used twice https://lucene.apache.org/solr/guide/7_1/distributed-requests.html once slave node that's where response writer yields "json" content and it upset aggregator node which is expect only javabin. I hardly can comment on rrw, it's probably used for responding separate files in distrib=false mode. You can start from describing why you need to create own response writer. On Tue, Apr 17, 2018 at 7:02 PM, Lee Carrollwrote: > Ok. My expectation was the response writer would not be used until the > final serialization of the result. If my response writer breaks the > response writer contract, exactly the way rawResponseWriter does and just > out puts a filed value how does that work? Does rawResponseWriter support > cloud mode? > > > > On 17 April 2018 at 15:55, Mikhail Khludnev wrote: > > > That's what should happen. > > > > Expected mime type application/octet-stream but got application/json. > > > > Distributed search coordinator expect to merge slave responses in javabin > > format. But slave's wt indicated json. > > As far as I know only javabin might be used to distributed search > > underneath. Coordinator itself might yield json. > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll < > lee.a.carr...@googlemail.com > > > > > wrote: > > > > > Sure > > > > > > with 1 shard 1 replica this request works fine > > > > > >1. Request URL: > > >http://localhost:8983/solr/images/image?q=id:1 > > >2. Request Method: > > >GET > > >3. Status Code: > > >200 OK > > > > > > logs are clean > > > > > > with 2 shards 2 replicas the same request fails and in the logs > > > > > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7 > > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore; > > > [images_shard2_replica_n4] webapp=/solr path=/image > > > params={df=text=false=/image=id=score& > > > shards.purpose=4=0=true= > > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/ > > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/ > > > =10=2=id:1=1523971232039=true=javabin} > > > hits=0 status=0 QTime=0 > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 > > > x:images_shard1_replica_n1] org.apache.solr.common.SolrException; > > > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: > > > Error > > > from server at http://10.224.30.207:8983/solr/images_shard2_replica_n4 > : > > > Expected mime type application/octet-stream but got application/json. > > > at > > > org.apache.solr.client.solrj.impl.HttpSolrClient. > > > executeMethod(HttpSolrClient.java:607) > > > at > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > HttpSolrClient.java:255) > > > at > > > org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > > HttpSolrClient.java:244) > > > at > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > doRequest(LBHttpSolrClient.java:483) > > > at > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request( > > > LBHttpSolrClient.java:413) > > > at > > > org.apache.solr.handler.component.HttpShardHandlerFactory. > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273) > > > at > > > org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0( > > > HttpShardHandler.java:175) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > at java.util.concurrent.Executors$RunnableAdapter. > > call(Executors.java:511) > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > > at > > > com.codahale.metrics.InstrumentedExecutorService$ > > InstrumentedRunnable.run( > > > InstrumentedExecutorService.java:176) > > > at > > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor. > > > lambda$execute$0(ExecutorUtil.java:188) > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker( > > > ThreadPoolExecutor.java:1142) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > ThreadPoolExecutor.java:617) > > > at java.lang.Thread.run(Thread.java:745) > > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3 > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore; > > > [images_shard1_replica_n1] webapp=/solr path=/image params={q=id:1} > > > status=200 QTime=17 > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 > > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore; > > > [images_shard1_replica_n1] webapp=/solr path=/image > > > params={df=text=false=/image=id=score& > > > shards.purpose=4=0=true= > > > http://10.224.30.207:8983/solr/images_shard1_replica_n1/ > > > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/ > > > =10=2=id:1=1523971232039=true=javabin} > > > hits=1 status=0 QTime=2 > > > > > > > > > I've implemented getcontenttype simply as > > > > > > @Override > > > public String getContentType(SolrQueryRequest request, > > > SolrQueryResponse response) { > >
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Ok. My expectation was the response writer would not be used until the final serialization of the result. If my response writer breaks the response writer contract, exactly the way rawResponseWriter does and just out puts a filed value how does that work? Does rawResponseWriter support cloud mode? On 17 April 2018 at 15:55, Mikhail Khludnevwrote: > That's what should happen. > > Expected mime type application/octet-stream but got application/json. > > Distributed search coordinator expect to merge slave responses in javabin > format. But slave's wt indicated json. > As far as I know only javabin might be used to distributed search > underneath. Coordinator itself might yield json. > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll > > wrote: > > > Sure > > > > with 1 shard 1 replica this request works fine > > > >1. Request URL: > >http://localhost:8983/solr/images/image?q=id:1 > >2. Request Method: > >GET > >3. Status Code: > >200 OK > > > > logs are clean > > > > with 2 shards 2 replicas the same request fails and in the logs > > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7 > > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore; > > [images_shard2_replica_n4] webapp=/solr path=/image > > params={df=text=false=/image=id=score& > > shards.purpose=4=0=true= > > http://10.224.30.207:8983/solr/images_shard2_replica_n4/ > > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/ > > =10=2=id:1=1523971232039=true=javabin} > > hits=0 status=0 QTime=0 > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 > > x:images_shard1_replica_n1] org.apache.solr.common.SolrException; > > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: > > Error > > from server at http://10.224.30.207:8983/solr/images_shard2_replica_n4: > > Expected mime type application/octet-stream but got application/json. > > at > > org.apache.solr.client.solrj.impl.HttpSolrClient. > > executeMethod(HttpSolrClient.java:607) > > at > > org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > HttpSolrClient.java:255) > > at > > org.apache.solr.client.solrj.impl.HttpSolrClient.request( > > HttpSolrClient.java:244) > > at > > org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > doRequest(LBHttpSolrClient.java:483) > > at > > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request( > > LBHttpSolrClient.java:413) > > at > > org.apache.solr.handler.component.HttpShardHandlerFactory. > > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273) > > at > > org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0( > > HttpShardHandler.java:175) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > com.codahale.metrics.InstrumentedExecutorService$ > InstrumentedRunnable.run( > > InstrumentedExecutorService.java:176) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor. > > lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker( > > ThreadPoolExecutor.java:1142) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > > ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3 > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore; > > [images_shard1_replica_n1] webapp=/solr path=/image params={q=id:1} > > status=200 QTime=17 > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 > > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore; > > [images_shard1_replica_n1] webapp=/solr path=/image > > params={df=text=false=/image=id=score& > > shards.purpose=4=0=true= > > http://10.224.30.207:8983/solr/images_shard1_replica_n1/ > > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/ > > =10=2=id:1=1523971232039=true=javabin} > > hits=1 status=0 QTime=2 > > > > > > I've implemented getcontenttype simply as > > > > @Override > > public String getContentType(SolrQueryRequest request, > > SolrQueryResponse response) { > > > > return "application/json;charset=utf-8"; > > } > > > > > > > > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev wrote: > > > > > Lee, > > > It's worth to send a stacktrace for such kind of inquiries. > > > I guess it goes from QueryComponent.mergeIds() or so. Shard response > > should > > > contains from schema.xml field. > > > I encounter something like this while troubleshooting > > > https://lucene.apache.org/solr/guide/6_6/transforming- > > > result-documents.html#TransformingResultDocuments- > > > CoresandCollectionsinSolrCloud > > > > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll < > > lee.a.carr...@googlemail.com > > > > > > > wrote: > > > > > > > I've created a custom response
Re: custom response writer which extends RawResponseWriter fails when shards > 1
That's what should happen. Expected mime type application/octet-stream but got application/json. Distributed search coordinator expect to merge slave responses in javabin format. But slave's wt indicated json. As far as I know only javabin might be used to distributed search underneath. Coordinator itself might yield json. On Tue, Apr 17, 2018 at 4:23 PM, Lee Carrollwrote: > Sure > > with 1 shard 1 replica this request works fine > >1. Request URL: >http://localhost:8983/solr/images/image?q=id:1 >2. Request Method: >GET >3. Status Code: >200 OK > > logs are clean > > with 2 shards 2 replicas the same request fails and in the logs > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7 > x:images_shard2_replica_n4] org.apache.solr.core.SolrCore; > [images_shard2_replica_n4] webapp=/solr path=/image > params={df=text=false=/image=id=score& > shards.purpose=4=0=true= > http://10.224.30.207:8983/solr/images_shard2_replica_n4/ > |http://10.224.30.207:7574/solr/images_shard2_replica_n6/ > =10=2=id:1=1523971232039=true=javabin} > hits=0 status=0 QTime=0 > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 > x:images_shard1_replica_n1] org.apache.solr.common.SolrException; > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: > Error > from server at http://10.224.30.207:8983/solr/images_shard2_replica_n4: > Expected mime type application/octet-stream but got application/json. > at > org.apache.solr.client.solrj.impl.HttpSolrClient. > executeMethod(HttpSolrClient.java:607) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request( > HttpSolrClient.java:255) > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request( > HttpSolrClient.java:244) > at > org.apache.solr.client.solrj.impl.LBHttpSolrClient. > doRequest(LBHttpSolrClient.java:483) > at > org.apache.solr.client.solrj.impl.LBHttpSolrClient.request( > LBHttpSolrClient.java:413) > at > org.apache.solr.handler.component.HttpShardHandlerFactory. > makeLoadBalancedRequest(HttpShardHandlerFactory.java:273) > at > org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0( > HttpShardHandler.java:175) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run( > InstrumentedExecutorService.java:176) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor. > lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3 > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore; > [images_shard1_replica_n1] webapp=/solr path=/image params={q=id:1} > status=200 QTime=17 > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 > x:images_shard1_replica_n1] org.apache.solr.core.SolrCore; > [images_shard1_replica_n1] webapp=/solr path=/image > params={df=text=false=/image=id=score& > shards.purpose=4=0=true= > http://10.224.30.207:8983/solr/images_shard1_replica_n1/ > |http://10.224.30.207:7574/solr/images_shard1_replica_n2/ > =10=2=id:1=1523971232039=true=javabin} > hits=1 status=0 QTime=2 > > > I've implemented getcontenttype simply as > > @Override > public String getContentType(SolrQueryRequest request, > SolrQueryResponse response) { > > return "application/json;charset=utf-8"; > } > > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev wrote: > > > Lee, > > It's worth to send a stacktrace for such kind of inquiries. > > I guess it goes from QueryComponent.mergeIds() or so. Shard response > should > > contains from schema.xml field. > > I encounter something like this while troubleshooting > > https://lucene.apache.org/solr/guide/6_6/transforming- > > result-documents.html#TransformingResultDocuments- > > CoresandCollectionsinSolrCloud > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll < > lee.a.carr...@googlemail.com > > > > > wrote: > > > > > I've created a custom response writer which extends RawResponseWriter. > > The > > > basic operation is to output a single field value from the top matching > > doc > > > as the entire response. This works when shards = 1 but fails when > shards > > > are greater than 1. > > > > > > I throw an error if the field in question is missing from the top doc. > > This > > > happens when individual shards are being searched and only id and score > > are > > > returned. I'm sure I've committed a basic error. > > > > > > Lee C > > > > > > > > > > > -- > > Sincerely yours > > Mikhail Khludnev > > > -- Sincerely yours Mikhail Khludnev
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Sure with 1 shard 1 replica this request works fine 1. Request URL: http://localhost:8983/solr/images/image?q=id:1 2. Request Method: GET 3. Status Code: 200 OK logs are clean with 2 shards 2 replicas the same request fails and in the logs INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 r:core_node7 x:images_shard2_replica_n4] org.apache.solr.core.SolrCore; [images_shard2_replica_n4] webapp=/solr path=/image params={df=text=false=/image=id=score=4=0=true= http://10.224.30.207:8983/solr/images_shard2_replica_n4/|http://10.224.30.207:7574/solr/images_shard2_replica_n6/=10=2=id:1=1523971232039=true=javabin} hits=0 status=0 QTime=0 ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 x:images_shard1_replica_n1] org.apache.solr.common.SolrException; org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://10.224.30.207:8983/solr/images_shard2_replica_n4: Expected mime type application/octet-stream but got application/json. at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:607) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483) at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413) at org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:273) at org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:175) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 r:core_node3 x:images_shard1_replica_n1] org.apache.solr.core.SolrCore; [images_shard1_replica_n1] webapp=/solr path=/image params={q=id:1} status=200 QTime=17 INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 r:core_node3 x:images_shard1_replica_n1] org.apache.solr.core.SolrCore; [images_shard1_replica_n1] webapp=/solr path=/image params={df=text=false=/image=id=score=4=0=true= http://10.224.30.207:8983/solr/images_shard1_replica_n1/|http://10.224.30.207:7574/solr/images_shard1_replica_n2/=10=2=id:1=1523971232039=true=javabin} hits=1 status=0 QTime=2 I've implemented getcontenttype simply as @Override public String getContentType(SolrQueryRequest request, SolrQueryResponse response) { return "application/json;charset=utf-8"; } On 16 April 2018 at 17:37, Mikhail Khludnevwrote: > Lee, > It's worth to send a stacktrace for such kind of inquiries. > I guess it goes from QueryComponent.mergeIds() or so. Shard response should > contains from schema.xml field. > I encounter something like this while troubleshooting > https://lucene.apache.org/solr/guide/6_6/transforming- > result-documents.html#TransformingResultDocuments- > CoresandCollectionsinSolrCloud > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll > > wrote: > > > I've created a custom response writer which extends RawResponseWriter. > The > > basic operation is to output a single field value from the top matching > doc > > as the entire response. This works when shards = 1 but fails when shards > > are greater than 1. > > > > I throw an error if the field in question is missing from the top doc. > This > > happens when individual shards are being searched and only id and score > are > > returned. I'm sure I've committed a basic error. > > > > Lee C > > > > > > -- > Sincerely yours > Mikhail Khludnev >
Re: custom response writer which extends RawResponseWriter fails when shards > 1
Lee, It's worth to send a stacktrace for such kind of inquiries. I guess it goes from QueryComponent.mergeIds() or so. Shard response should contains from schema.xml field. I encounter something like this while troubleshooting https://lucene.apache.org/solr/guide/6_6/transforming-result-documents.html#TransformingResultDocuments-CoresandCollectionsinSolrCloud On Mon, Apr 16, 2018 at 6:56 PM, Lee Carrollwrote: > I've created a custom response writer which extends RawResponseWriter. The > basic operation is to output a single field value from the top matching doc > as the entire response. This works when shards = 1 but fails when shards > are greater than 1. > > I throw an error if the field in question is missing from the top doc. This > happens when individual shards are being searched and only id and score are > returned. I'm sure I've committed a basic error. > > Lee C > -- Sincerely yours Mikhail Khludnev
custom response writer which extends RawResponseWriter fails when shards > 1
I've created a custom response writer which extends RawResponseWriter. The basic operation is to output a single field value from the top matching doc as the entire response. This works when shards = 1 but fails when shards are greater than 1. I throw an error if the field in question is missing from the top doc. This happens when individual shards are being searched and only id and score are returned. I'm sure I've committed a basic error. Lee C