Re: Sending compressed (gzip) UpdateRequest with SolrJ
Years ago, working on the Ultraseek spider, we did a bunch of tests on compressed HTTP. I expected it to be a big win, but the results were really inconclusive. Sometimes it was faster, sometimes it was slower. We left it turned off. It is an absolute win for serving already-compressed static content with Apache or whatever. For dynamic content, it will increase some amount of delay as stuff is compressed before sending. If the content already fits in one or two packets, it is just extra overhead. For really large data, it helps with transmission time, but the processing time for large data probably overwhelms the network time. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Jan 8, 2021, at 12:01 AM, Gael Jourdan-Weil > wrote: > > You're right Matthew. > > Jetty supports it for responses but for requests it doesn't seem to be the > default. > However I found a configuration not documented that needs to be set in the > GzipHandler for it to work: inflateBufferSize. > > For SolrJ it still hacky to send gzip requests, maybe easier to use a regular > http call.. > > --- > > De : matthew sporleder > Envoyé : jeudi 7 janvier 2021 16:43 > À : solr-user@lucene.apache.org > Objet : Re: Sending compressed (gzip) UpdateRequest with SolrJ > > jetty supports http gzip and I've added it to solr before in my own > installs (and submitted patches to do so by default to solr) but I > don't know about the handling for solrj. > > IME compression helps a little, sometimes a lot, and never hurts. > Even the admin interface benefits a lot from regular old http gzip > > On Thu, Jan 7, 2021 at 8:03 AM Gael Jourdan-Weil > wrote: >> >> Answering to myself on this one. >> >> Solr uses Jetty 9.x which does not support compressed requests by itself >> meaning, the application behind Jetty (that is Solr) has to decompress by >> itself which is not the case for now. >> Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with >> cURL for instance) is not possible for now. >> >> Seems quite surprising to me though. >> >> - >> >> Hello, >> >> I was wondering if someone ever had the need to send compressed (gzip) >> update requests (adding/deleting documents), especially using SolrJ. >> >> Somehow I expected it to be done by default, but didn't find any >> documentation about it and when looking at the code it seems there is no >> option to do it. Or is javabin compressed by default? >> - >> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49 >> - >> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 >> (if not using Javabin) >> - >> https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587 >> >> By the way, is there any documentation about javabin? I could only find one >> on the "old wiki". >> >> Thanks, >> Gaël
RE: Sending compressed (gzip) UpdateRequest with SolrJ
You're right Matthew. Jetty supports it for responses but for requests it doesn't seem to be the default. However I found a configuration not documented that needs to be set in the GzipHandler for it to work: inflateBufferSize. For SolrJ it still hacky to send gzip requests, maybe easier to use a regular http call.. --- De : matthew sporleder Envoyé : jeudi 7 janvier 2021 16:43 À : solr-user@lucene.apache.org Objet : Re: Sending compressed (gzip) UpdateRequest with SolrJ jetty supports http gzip and I've added it to solr before in my own installs (and submitted patches to do so by default to solr) but I don't know about the handling for solrj. IME compression helps a little, sometimes a lot, and never hurts. Even the admin interface benefits a lot from regular old http gzip On Thu, Jan 7, 2021 at 8:03 AM Gael Jourdan-Weil wrote: > > Answering to myself on this one. > > Solr uses Jetty 9.x which does not support compressed requests by itself > meaning, the application behind Jetty (that is Solr) has to decompress by > itself which is not the case for now. > Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with > cURL for instance) is not possible for now. > > Seems quite surprising to me though. > > - > > Hello, > > I was wondering if someone ever had the need to send compressed (gzip) update > requests (adding/deleting documents), especially using SolrJ. > > Somehow I expected it to be done by default, but didn't find any > documentation about it and when looking at the code it seems there is no > option to do it. Or is javabin compressed by default? > - > https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49 > - > https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 > (if not using Javabin) > - > https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587 > > By the way, is there any documentation about javabin? I could only find one > on the "old wiki". > > Thanks, > Gaël
Re: Sending compressed (gzip) UpdateRequest with SolrJ
jetty supports http gzip and I've added it to solr before in my own installs (and submitted patches to do so by default to solr) but I don't know about the handling for solrj. IME compression helps a little, sometimes a lot, and never hurts. Even the admin interface benefits a lot from regular old http gzip On Thu, Jan 7, 2021 at 8:03 AM Gael Jourdan-Weil wrote: > > Answering to myself on this one. > > Solr uses Jetty 9.x which does not support compressed requests by itself > meaning, the application behind Jetty (that is Solr) has to decompress by > itself which is not the case for now. > Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with > cURL for instance) is not possible for now. > > Seems quite surprising to me though. > > - > > Hello, > > I was wondering if someone ever had the need to send compressed (gzip) update > requests (adding/deleting documents), especially using SolrJ. > > Somehow I expected it to be done by default, but didn't find any > documentation about it and when looking at the code it seems there is no > option to do it. Or is javabin compressed by default? > - > https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49 > - > https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 > (if not using Javabin) > - > https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587 > > By the way, is there any documentation about javabin? I could only find one > on the "old wiki". > > Thanks, > Gaël
RE: Sending compressed (gzip) UpdateRequest with SolrJ
Answering to myself on this one. Solr uses Jetty 9.x which does not support compressed requests by itself meaning, the application behind Jetty (that is Solr) has to decompress by itself which is not the case for now. Thus even without using SolrJ, sending XML compressed in GZIP to Solr (with cURL for instance) is not possible for now. Seems quite surprising to me though. - Hello, I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ. Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default? - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49 - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin) - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587 By the way, is there any documentation about javabin? I could only find one on the "old wiki". Thanks, Gaël
Sending compressed (gzip) UpdateRequest with SolrJ
Hello, I was wondering if someone ever had the need to send compressed (gzip) update requests (adding/deleting documents), especially using SolrJ. Somehow I expected it to be done by default, but didn't find any documentation about it and when looking at the code it seems there is no option to do it. Or is javabin compressed by default? - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BinaryRequestWriter.java#L49 - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/RequestWriter.java#L55 (if not using Javabin) - https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/client/solrj/impl/Http2SolrClient.java#L587 By the way, is there any documentation about javabin? I could only find one on the "old wiki". Thanks, Gaël
