what if you put it into "defaults"? On Thu, Apr 19, 2018 at 8:42 PM, Lee Carroll <lee.a.carr...@googlemail.com> wrote:
> Hi, > > I rewrote all of my tests to use SolrCloudTestCase rather than > SolrTestCaseJ4 > and was able to replicate the responsewriter issue and debug with a sharded > collection. It turned out the issue was not with my response writer really > but rather my config. > > <requestHandler name="/content" class="solr.SearchHandler"> > .... > > <lst name="invariants"> > <str name="wt">content</str> > </lst> > > </requestHandler> > > In cloud mode having wt as an invariant breaks the collation of results > from shards. Now I'm sure this is a common mistake which I've repeated > (blush) but I do sort of want to actually implement my request handler in > this way. Is their a way to have a request handler support a single > response writer but still work in cloud mode ? > > Could this be considered a bug ? > > Lee C > > On 18 April 2018 at 13:13, Mikhail Khludnev <m...@apache.org> wrote: > > > Injecting headers might require deeper customisation up to establishing > own > > filter or so. > > Speaking regarding your own WT, there might be some issues because > usually > > it's not a big deal to use one wt for responding user query like (wt=csv) > > and wt=javabin in internal communication between aggregator and slaves > like > > it happens in wt=csv query. > > > > On Wed, Apr 18, 2018 at 2:19 PM, Lee Carroll < > lee.a.carr...@googlemail.com > > > > > wrote: > > > > > Inventive. I need to control content-type of the response from the > > document > > > field value. I have the actual content field and the content-type field > > to > > > use configured in the response writer. I've just noticed that the xslt > > > transformer allows you to do this but not controlled by document > values. > > I > > > may also need to set some headers based on content-type and perhaps > > content > > > size, accept ranges comes to mind. Although I might be getting ahead of > > > myself. > > > > > > > > > > > > On 18 April 2018 at 12:05, Mikhail Khludnev <m...@apache.org> wrote: > > > > > > > well .. > > > > what if > > > > http://localhost:8983/solr/images/select?fl=content&q=id: > > > 1&start=1&wt=csv& > > > > csv.separator=&csv.encapsulator&csv.null=null > > > > ? > > > > > > > > On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll < > > > lee.a.carr...@googlemail.com > > > > > > > > > wrote: > > > > > > > > > sorry cut n paste error i'd get > > > > > > > > > > { > > > > > "responseHeader":{ > > > > > "zkConnected":true, > > > > > "status":0, > > > > > "QTime":0, > > > > > "params":{ > > > > > "q":"*:*", > > > > > "fl":"content", > > > > > "rows":"1"}}, > > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > > { > > > > > "content":"my-content-value"}] > > > > > }} > > > > > > > > > > > > > > > but you get my point > > > > > > > > > > > > > > > > > > > > On 18 April 2018 at 11:13, Lee Carroll < > lee.a.carr...@googlemail.com > > > > > > > > wrote: > > > > > > > > > > > for http://localhost:8983/solr/images/select?fl=content&q=id: > > > 1&start=1 > > > > > > > > > > > > I'd get > > > > > > > > > > > > { > > > > > > "responseHeader":{ > > > > > > "zkConnected":true, > > > > > > "status":0, > > > > > > "QTime":1, > > > > > > "params":{ > > > > > > "q":"*:*", > > > > > > "_":"1524046333220"}}, > > > > > > "response":{"numFound":1,"start":0,"docs":[ > > > > > > { > > > > > > "id":"1", > > > > > > "content":"my-content-value", > > > > > > "*content-type*":"text/plain"}] > > > > > > }} > > > > > > > > > > > > when i want > > > > > > > > > > > > my-content-value > > > > > > > > > > > > > > > > > > > > > > > > On 18 April 2018 at 10:55, Mikhail Khludnev <m...@apache.org> > > wrote: > > > > > > > > > > > >> Lee, from this description I don see why it can't be addressed > by > > > > > fl,rows > > > > > >> params. What makes it different form the typical Solr usage? > > > > > >> > > > > > >> > > > > > >> On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll < > > > > > >> lee.a.carr...@googlemail.com> > > > > > >> wrote: > > > > > >> > > > > > >> > Sure, we want to return a single field's value for the top > > > matching > > > > > >> > document for a given query. Bare content rather than a full > > search > > > > > >> result > > > > > >> > listing. > > > > > >> > > > > > > >> > To be concrete: > > > > > >> > > > > > > >> > For a schema of fields id [unique key], > > > > content[stored],content-type[ > > > > > >> > stored] > > > > > >> > For a request: > > > > > >> > > > > > > >> > 1. Request URL: > > > > > >> > https://localhost/solr/content?q=id:1 > > > > > >> > 2. Request Method: > > > > > >> > GET > > > > > >> > > > > > > >> > We get a response > > > > > >> > HTTP/1.1 200 OK Content-Length: 16261 Content-Type: > > [content-type > > > > > value] > > > > > >> > > > > > > >> > and the body to be the raw value of content > > > > > >> > > > > > > >> > In short clients consume directly the most relevant "content" > > > > returned > > > > > >> from > > > > > >> > solr queries they construct. > > > > > >> > > > > > > >> > Naively I've implemented a subclass of RawResponseWriter which > > > takes > > > > > the > > > > > >> > first docs values and adds them to the appended "content" > > stream. > > > > > >> Should I > > > > > >> > selectively add the content stream depending on if this is the > > > final > > > > > >> > aggregation of cloud results (and provide a base class writer > to > > > act > > > > > if > > > > > >> > not), if so how do I know its the final aggregation. Or is > > adding > > > > the > > > > > >> > content stream within the response writer a bad idea. Should > > that > > > be > > > > > >> being > > > > > >> > added to the response somewhere else? > > > > > >> > > > > > > >> > Failing all of the above is asking about response writer an X > / > > Y > > > > > >> problem. > > > > > >> > Is their a better way to achieve the above. I'd looked at > > > > transforming > > > > > >> > response xml but that seemed not to offer a complete bare > slate. > > > > > >> > > > > > > >> > Cheers Lee C > > > > > >> > > > > > > >> > > > > > > >> > On 17 April 2018 at 21:36, Mikhail Khludnev <m...@apache.org> > > > > wrote: > > > > > >> > > > > > > >> > > In distributed search response writer is used twice > > > > > >> > > https://lucene.apache.org/solr/guide/7_1/distributed- > > > > requests.html > > > > > >> > > once slave node that's where response writer yields "json" > > > content > > > > > >> and it > > > > > >> > > upset aggregator node which is expect only javabin. > > > > > >> > > I hardly can comment on rrw, it's probably used for > responding > > > > > >> separate > > > > > >> > > files in distrib=false mode. > > > > > >> > > You can start from describing why you need to create own > > > response > > > > > >> writer. > > > > > >> > > > > > > > >> > > On Tue, Apr 17, 2018 at 7:02 PM, Lee Carroll < > > > > > >> > lee.a.carr...@googlemail.com > > > > > >> > > > > > > > > >> > > wrote: > > > > > >> > > > > > > > >> > > > Ok. My expectation was the response writer would not be > used > > > > until > > > > > >> the > > > > > >> > > > final serialization of the result. If my response writer > > > breaks > > > > > the > > > > > >> > > > response writer contract, exactly the way > rawResponseWriter > > > does > > > > > and > > > > > >> > just > > > > > >> > > > out puts a filed value how does that work? Does > > > > rawResponseWriter > > > > > >> > support > > > > > >> > > > cloud mode? > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > On 17 April 2018 at 15:55, Mikhail Khludnev < > > m...@apache.org> > > > > > >> wrote: > > > > > >> > > > > > > > > >> > > > > That's what should happen. > > > > > >> > > > > > > > > > >> > > > > Expected mime type application/octet-stream but got > > > > > >> application/json. > > > > > >> > > > > > > > > > >> > > > > Distributed search coordinator expect to merge slave > > > responses > > > > > in > > > > > >> > > javabin > > > > > >> > > > > format. But slave's wt indicated json. > > > > > >> > > > > As far as I know only javabin might be used to > distributed > > > > > search > > > > > >> > > > > underneath. Coordinator itself might yield json. > > > > > >> > > > > > > > > > >> > > > > On Tue, Apr 17, 2018 at 4:23 PM, Lee Carroll < > > > > > >> > > > lee.a.carr...@googlemail.com > > > > > >> > > > > > > > > > > >> > > > > wrote: > > > > > >> > > > > > > > > > >> > > > > > Sure > > > > > >> > > > > > > > > > > >> > > > > > with 1 shard 1 replica this request works fine > > > > > >> > > > > > > > > > > >> > > > > > 1. Request URL: > > > > > >> > > > > > http://localhost:8983/solr/images/image?q=id:1 > > > > > >> > > > > > 2. Request Method: > > > > > >> > > > > > GET > > > > > >> > > > > > 3. Status Code: > > > > > >> > > > > > 200 OK > > > > > >> > > > > > > > > > > >> > > > > > logs are clean > > > > > >> > > > > > > > > > > >> > > > > > with 2 shards 2 replicas the same request fails and in > > the > > > > > logs > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > INFO - 2018-04-17 13:20:32.052; [c:images s:shard2 > > > > > r:core_node7 > > > > > >> > > > > > x:images_shard2_replica_n4] > > org.apache.solr.core.SolrCore; > > > > > >> > > > > > [images_shard2_replica_n4] webapp=/solr path=/image > > > > > >> > > > > > params={df=text&distrib=false& > qt=/image&fl=id&fl=score& > > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url= > > > > > >> > > > > > http://10.224.30.207:8983/ > > solr/images_shard2_replica_n4/ > > > > > >> > > > > > |http://10.224.30.207:7574/ > > solr/images_shard2_replica_n6/ > > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW= > > > > 1523971232039&isShard=true&wt= > > > > > >> > javabin} > > > > > >> > > > > > hits=0 status=0 QTime=0 > > > > > >> > > > > > ERROR - 2018-04-17 13:20:32.055; [c:images s:shard1 > > > > > r:core_node3 > > > > > >> > > > > > x:images_shard1_replica_n1] > > org.apache.solr.common.SolrExc > > > > > >> eption; > > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient$ > > > > > >> > > RemoteSolrException: > > > > > >> > > > > > Error > > > > > >> > > > > > from server at http://10.224.30.207:8983/ > > > > > >> > > solr/images_shard2_replica_n4 > > > > > >> > > > : > > > > > >> > > > > > Expected mime type application/octet-stream but got > > > > > >> > application/json. > > > > > >> > > > > > at > > > > > >> > > > > > org.apache.solr.client.solrj.impl.HttpSolrClient. > > > > > >> > > > > > executeMethod(HttpSolrClient.java:607) > > > > > >> > > > > > at > > > > > >> > > > > > org.apache.solr.client.solrj. > > impl.HttpSolrClient.request( > > > > > >> > > > > > HttpSolrClient.java:255) > > > > > >> > > > > > at > > > > > >> > > > > > org.apache.solr.client.solrj. > > impl.HttpSolrClient.request( > > > > > >> > > > > > HttpSolrClient.java:244) > > > > > >> > > > > > at > > > > > >> > > > > > org.apache.solr.client.solrj.impl.LBHttpSolrClient. > > > > > >> > > > > > doRequest(LBHttpSolrClient.java:483) > > > > > >> > > > > > at > > > > > >> > > > > > org.apache.solr.client.solrj. > > > impl.LBHttpSolrClient.request( > > > > > >> > > > > > LBHttpSolrClient.java:413) > > > > > >> > > > > > at > > > > > >> > > > > > org.apache.solr.handler.component. > > > HttpShardHandlerFactory. > > > > > >> > > > > > makeLoadBalancedRequest(HttpShardHandlerFactory.java: > > 273) > > > > > >> > > > > > at > > > > > >> > > > > > org.apache.solr.handler.component.HttpShardHandler. > > > > > >> > lambda$submit$0( > > > > > >> > > > > > HttpShardHandler.java:175) > > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask. > > > java:266) > > > > > >> > > > > > at java.util.concurrent.Executors$RunnableAdapter. > > > > > >> > > > > call(Executors.java:511) > > > > > >> > > > > > at java.util.concurrent.FutureTask.run(FutureTask. > > > java:266) > > > > > >> > > > > > at > > > > > >> > > > > > com.codahale.metrics.InstrumentedExecutorService$ > > > > > >> > > > > InstrumentedRunnable.run( > > > > > >> > > > > > InstrumentedExecutorService.java:176) > > > > > >> > > > > > at > > > > > >> > > > > > org.apache.solr.common.util.ExecutorUtil$ > > > > > >> > MDCAwareThreadPoolExecutor. > > > > > >> > > > > > lambda$execute$0(ExecutorUtil.java:188) > > > > > >> > > > > > at > > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( > > > > > >> > > > > > ThreadPoolExecutor.java:1142) > > > > > >> > > > > > at > > > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > > > >> > > > > > ThreadPoolExecutor.java:617) > > > > > >> > > > > > at java.lang.Thread.run(Thread.java:745) > > > > > >> > > > > > > > > > > >> > > > > > INFO - 2018-04-17 13:20:32.056; [c:images s:shard1 > > > > > r:core_node3 > > > > > >> > > > > > x:images_shard1_replica_n1] > > org.apache.solr.core.SolrCore; > > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image > > > > > >> > params={q=id:1} > > > > > >> > > > > > status=200 QTime=17 > > > > > >> > > > > > INFO - 2018-04-17 13:20:32.055; [c:images s:shard1 > > > > > r:core_node3 > > > > > >> > > > > > x:images_shard1_replica_n1] > > org.apache.solr.core.SolrCore; > > > > > >> > > > > > [images_shard1_replica_n1] webapp=/solr path=/image > > > > > >> > > > > > params={df=text&distrib=false& > qt=/image&fl=id&fl=score& > > > > > >> > > > > > shards.purpose=4&start=0&fsv=true&shard.url= > > > > > >> > > > > > http://10.224.30.207:8983/ > > solr/images_shard1_replica_n1/ > > > > > >> > > > > > |http://10.224.30.207:7574/ > > solr/images_shard1_replica_n2/ > > > > > >> > > > > > &rows=10&version=2&q=id:1&NOW= > > > > 1523971232039&isShard=true&wt= > > > > > >> > javabin} > > > > > >> > > > > > hits=1 status=0 QTime=2 > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > I've implemented getcontenttype simply as > > > > > >> > > > > > > > > > > >> > > > > > @Override > > > > > >> > > > > > public String getContentType(SolrQueryRequest > request, > > > > > >> > > > > > SolrQueryResponse response) { > > > > > >> > > > > > > > > > > >> > > > > > return "application/json;charset=utf-8"; > > > > > >> > > > > > } > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > On 16 April 2018 at 17:37, Mikhail Khludnev < > > > > m...@apache.org> > > > > > >> > wrote: > > > > > >> > > > > > > > > > > >> > > > > > > Lee, > > > > > >> > > > > > > It's worth to send a stacktrace for such kind of > > > > inquiries. > > > > > >> > > > > > > I guess it goes from QueryComponent.mergeIds() or > so. > > > > Shard > > > > > >> > > response > > > > > >> > > > > > should > > > > > >> > > > > > > contains <uniqueKey> from schema.xml field. > > > > > >> > > > > > > I encounter something like this while > troubleshooting > > > > > >> > > > > > > https://lucene.apache.org/ > > solr/guide/6_6/transforming- > > > > > >> > > > > > > result-documents.html#TransformingResultDocuments- > > > > > >> > > > > > > CoresandCollectionsinSolrCloud > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > On Mon, Apr 16, 2018 at 6:56 PM, Lee Carroll < > > > > > >> > > > > > lee.a.carr...@googlemail.com > > > > > >> > > > > > > > > > > > > >> > > > > > > wrote: > > > > > >> > > > > > > > > > > > >> > > > > > > > I've created a custom response writer which > extends > > > > > >> > > > > RawResponseWriter. > > > > > >> > > > > > > The > > > > > >> > > > > > > > basic operation is to output a single field value > > from > > > > the > > > > > >> top > > > > > >> > > > > matching > > > > > >> > > > > > > doc > > > > > >> > > > > > > > as the entire response. This works when shards = 1 > > but > > > > > fails > > > > > >> > when > > > > > >> > > > > > shards > > > > > >> > > > > > > > are greater than 1. > > > > > >> > > > > > > > > > > > > >> > > > > > > > I throw an error if the field in question is > missing > > > > from > > > > > >> the > > > > > >> > top > > > > > >> > > > > doc. > > > > > >> > > > > > > This > > > > > >> > > > > > > > happens when individual shards are being searched > > and > > > > only > > > > > >> id > > > > > >> > and > > > > > >> > > > > score > > > > > >> > > > > > > are > > > > > >> > > > > > > > returned. I'm sure I've committed a basic error. > > > > > >> > > > > > > > > > > > > >> > > > > > > > Lee C > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > -- > > > > > >> > > > > > > Sincerely yours > > > > > >> > > > > > > Mikhail Khludnev > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > -- > > > > > >> > > > > Sincerely yours > > > > > >> > > > > Mikhail Khludnev > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > -- > > > > > >> > > Sincerely yours > > > > > >> > > Mikhail Khludnev > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> Sincerely yours > > > > > >> Mikhail Khludnev > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Sincerely yours > > > > Mikhail Khludnev > > > > > > > > > > > > > > > -- > > Sincerely yours > > Mikhail Khludnev > > > -- Sincerely yours Mikhail Khludnev
