Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-22 Thread Chris Hostetter
: Subject: Re: custom response writer which extends RawResponseWriter fails when : shards > 1 : : Hi, : I've ended up processing the doclist in the response twice. Once in the : write method and once in getContent. Its a bit inefficient but i'm only : looking at top doc each time so

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-22 Thread Lee Carroll
iter if/when you see that this is an "isShard=true" request? >> >> >> >> : Date: Thu, 19 Apr 2018 18:42:58 +0100 >> : From: Lee Carroll >> : Reply-To: solr-user@lucene.apache.org >> : To: solr-user@lucene.apache.org >> : Subject: Re: custom respons

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-22 Thread Lee Carroll
0100 > : From: Lee Carroll > : Reply-To: solr-user@lucene.apache.org > : To: solr-user@lucene.apache.org > : Subject: Re: custom response writer which extends RawResponseWriter > fails when > : shards > 1 > : > : Hi, > : > : I rewrote all of my tests to use Sol

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-20 Thread Chris Hostetter
: Subject: Re: custom response writer which extends RawResponseWriter fails when : shards > 1 : : Hi, : : I rewrote all of my tests to use SolrCloudTestCase rather than SolrTestCaseJ4 : and was able to replicate the responsewriter issue and debug with a sharded : collection. It turned out t

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-19 Thread Mikhail Khludnev
You can introduce own searchHandler with wt invariant. When aggreator request slaves it will use regular /select with default wt=javabin (it's condrolled by shards.qt, btw) Providing such comprehensive application logic on top of neat solrconfig.xml is not the best idea, though. On Thu, Apr 19, 20

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-19 Thread Lee Carroll
Default works. However in the case (which is maybe my case) where request handler implies a response type and really should be locked down to prevent abuse or error you could argue invariant is needed. I guess its also not very elegant having an arbitrary rule, no wt as invariant in cloud mode etc.

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-19 Thread Mikhail Khludnev
what if you put it into "defaults"? On Thu, Apr 19, 2018 at 8:42 PM, Lee Carroll wrote: > Hi, > > I rewrote all of my tests to use SolrCloudTestCase rather than > SolrTestCaseJ4 > and was able to replicate the responsewriter issue and debug with a sharded > collection. It turned out the issue wa

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-19 Thread Lee Carroll
Hi, I rewrote all of my tests to use SolrCloudTestCase rather than SolrTestCaseJ4 and was able to replicate the responsewriter issue and debug with a sharded collection. It turned out the issue was not with my response writer really but rather my config. content In clo

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-18 Thread Mikhail Khludnev
Injecting headers might require deeper customisation up to establishing own filter or so. Speaking regarding your own WT, there might be some issues because usually it's not a big deal to use one wt for responding user query like (wt=csv) and wt=javabin in internal communication between aggregator

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-18 Thread Lee Carroll
Inventive. I need to control content-type of the response from the document field value. I have the actual content field and the content-type field to use configured in the response writer. I've just noticed that the xslt transformer allows you to do this but not controlled by document values. I ma

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-18 Thread Mikhail Khludnev
well .. what if http://localhost:8983/solr/images/select?fl=content&q=id:1&start=1&wt=csv&; csv.separator=&csv.encapsulator&csv.null=null ? On Wed, Apr 18, 2018 at 1:18 PM, Lee Carroll wrote: > sorry cut n paste error i'd get > > { > "responseHeader":{ > "zkConnected":true, > "status":

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-18 Thread Lee Carroll
sorry cut n paste error i'd get { "responseHeader":{ "zkConnected":true, "status":0, "QTime":0, "params":{ "q":"*:*", "fl":"content", "rows":"1"}}, "response":{"numFound":1,"start":0,"docs":[ { "content":"my-content-value"}] }} but you get my

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-18 Thread Lee Carroll
for http://localhost:8983/solr/images/select?fl=content&q=id:1&start=1 I'd get { "responseHeader":{ "zkConnected":true, "status":0, "QTime":1, "params":{ "q":"*:*", "_":"1524046333220"}}, "response":{"numFound":1,"start":0,"docs":[ { "id":"1",

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-18 Thread Mikhail Khludnev
Lee, from this description I don see why it can't be addressed by fl,rows params. What makes it different form the typical Solr usage? On Wed, Apr 18, 2018 at 12:31 PM, Lee Carroll wrote: > Sure, we want to return a single field's value for the top matching > document for a given query. Bare co

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-18 Thread Lee Carroll
Sure, we want to return a single field's value for the top matching document for a given query. Bare content rather than a full search result listing. To be concrete: For a schema of fields id [unique key], content[stored],content-type[stored] For a request: 1. Request URL: https://localho

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-17 Thread Mikhail Khludnev
In distributed search response writer is used twice https://lucene.apache.org/solr/guide/7_1/distributed-requests.html once slave node that's where response writer yields "json" content and it upset aggregator node which is expect only javabin. I hardly can comment on rrw, it's probably used for re

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-17 Thread Lee Carroll
Ok. My expectation was the response writer would not be used until the final serialization of the result. If my response writer breaks the response writer contract, exactly the way rawResponseWriter does and just out puts a filed value how does that work? Does rawResponseWriter support cloud mode?

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-17 Thread Mikhail Khludnev
That's what should happen. Expected mime type application/octet-stream but got application/json. Distributed search coordinator expect to merge slave responses in javabin format. But slave's wt indicated json. As far as I know only javabin might be used to distributed search underneath. Coordinat

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-17 Thread Lee Carroll
Sure with 1 shard 1 replica this request works fine 1. Request URL: http://localhost:8983/solr/images/image?q=id:1 2. Request Method: GET 3. Status Code: 200 OK logs are clean with 2 shards 2 replicas the same request fails and in the logs INFO - 2018-04-17 13:20:32.052; [c

Re: custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-16 Thread Mikhail Khludnev
Lee, It's worth to send a stacktrace for such kind of inquiries. I guess it goes from QueryComponent.mergeIds() or so. Shard response should contains from schema.xml field. I encounter something like this while troubleshooting https://lucene.apache.org/solr/guide/6_6/transforming-result-documents.

custom response writer which extends RawResponseWriter fails when shards > 1

2018-04-16 Thread Lee Carroll
I've created a custom response writer which extends RawResponseWriter. The basic operation is to output a single field value from the top matching doc as the entire response. This works when shards = 1 but fails when shards are greater than 1. I throw an error if the field in question is missing f