subject:"Re\: Re\: Low untunable default FastWriter output buffer \- possible reason for slow single threaded data receiving from Solr on 1Gigabit\+ networks while scroll, search etc"

Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-14 Thread David Smiley

You can get parallelism by sharding -- use numShards=8 or whatever number
of CPUs you have.

In your performance analysis, you did not speak of memory; you spoke as if
there are only two factors at play -- CPU and Disk.  Compression of data on
disk is used in order to allow the OS's disk cache to cache as much as
possible.  In primary search use-cases (but not return-all-data), RAM
matters because the OS uses it to cache disk.  Primary search use-cases are
very random-IO oriented; not sequential (unlike read-all-data).  If you
were to manage to turn off stored value compression, I'm sure you would
accomplish your current goal but then, I suspect, hurt a primary use-case.

To pursue disabling compression further: you could create a Lucene
CompressionMode subclass that doesn't actually do any compression.  This is
actually rather easy to do; I've subclassed this before so I'm familiar
with the mechanics of it.  There is more integration work to do of course.
Ultimately Solr would have to be configured to use a custom Codec.  At
work, I like to say "we've plugged all the pluggables and created our own"
LOL.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Tue, Mar 14, 2023 at 4:54 PM Fikavec F  wrote:

> Thank you for working on the Solr performance issues raised here.
>LZ4 is a great solution, but let's look at how things are today. As far
> as I understand, uncompressed fields have been abandoned since version 4.1
> (early 2013). At that time, 15,000 RPM SAS disks produced 350 MB/s or
> 150-180 iops and had a volume of about 300 GB up to 600 GB. Modern server
> PCI-E 5.0  NVME disks promise us 13,000 MB/s (37 times faster than SAS) or
> 2,500,000 random read IOPS (13,888 times faster than SAS) and a volume of
> up to 15.36 TB (25 times more than SAS). However, the frequency of server
> processors has not increased, (for example, Xeon ® E3-1245 v3 2013) they
> remained at the level of 2.2-3.40 GHz (although they became multicore and
> more efficient). I did not see in htop that all cores were loaded during
> data transfer, which means Solr does not decompress LZ4 data into many
> streams, which means LZ4 at what speed on one 3.40 GHz processor core in
> 2013 in Solr worked, with about the same and works in 2023, but disks
> from-for which LZ4 was introduced, they became 37 times more powerful than
> SAS, 13888 more powerful than SAS, and 25 times more SAS in volume. In this
> scenario, can't LZ4 turn out to be a serious bottleneck of Solr performance
> these days, or maybe if everything is tested well, it's time to turn LZ4
> off again (or even disabled by default) or add a parallel operation mode on
> several cores to LZ4?
> Unfortunately, I couldn't reproduce the code from the article and create a
> working collection with compression disabled, I couldn't test anything. It
> is also interesting that the fields are stored in LZ4, then decompressed,
> recompressed by gzip and sent to the client - perhaps in some cases using
> solr they could be stored in gzip to be quickly given directly from the
> disk to the client.
>
>The second interesting question is that I continued to study the effect
> of the size of the fields on the data transfer rate. I created a collection
> and filled it with "for i in range(40,000,000) insert
> {'id':str(i),text_s:str(i)}" documents and found that on my 4.5GHz CPU Solr
> can process (return) only about 115 000 documents per second. It doesn't
> look fast - there may be a bottleneck in iterating over the set of
> documents returned from Solr (for example, if some expensive resource/class
> is created somewhere for each returned document instead of being reused)?
>
>It seems to me that there is a benefit in such discussions and this
> does not concern my scenario of using Solr in any way. The equipment has
> become faster, its working principle has changed, taking into account the
> age of the Solr project, it may well turn out that some basic things should
> be reconsidered, rethought, changing to multithreaded or change something
> else. I opened the discussion itself due to the fact that my installation
> of SOLR worked by almost 90% equally on very old equipment and a new one is
> 10 times more powerful. The results achieved so far are already remarkable,
> but I think there is still potential for improvement. For example, a simple
> code review of a old-written SmileResponseWriter code made it possible to
> identify a bug and find a place that potentially negatively affects
> performance, this maybe useful for future.
>
> Best Regards,
> - To
> unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional
> commands, e-mail: dev-h...@solr.apache.org

RE: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-14 Thread Fikavec F

Thank you for working on the Solr performance issues raised here.   LZ4 is a great solution, but let's look at how things are today. As far as I understand, uncompressed fields have been abandoned since version 4.1 (early 2013). At that time, 15,000 RPM SAS disks produced 350 MB/s or 150-180 iops and had a volume of about 300 GB up to 600 GB. Modern server PCI-E 5.0  NVME disks promise us 13,000 MB/s (37 times faster than SAS) or 2,500,000 random read IOPS (13,888 times faster than SAS) and a volume of up to 15.36 TB (25 times more than SAS). However, the frequency of server processors has not increased, (for example, Xeon ® E3-1245 v3 2013) they remained at the level of 2.2-3.40 GHz (although they became multicore and more efficient). I did not see in htop that all cores were loaded during data transfer, which means Solr does not decompress LZ4 data into many streams, which means LZ4 at what speed on one 3.40 GHz processor core in 2013 in Solr worked, with about the same and works in 2023, but disks from-for which LZ4 was introduced, they became 37 times more powerful than SAS, 13888 more powerful than SAS, and 25 times more SAS in volume. In this scenario, can't LZ4 turn out to be a serious bottleneck of Solr performance these days, or maybe if everything is tested well, it's time to turn LZ4 off again (or even disabled by default) or add a parallel operation mode on several cores to LZ4?Unfortunately, I couldn't reproduce the code from the article and create a working collection with compression disabled, I couldn't test anything. It is also interesting that the fields are stored in LZ4, then decompressed, recompressed by gzip and sent to the client - perhaps in some cases using solr they could be stored in gzip to be quickly given directly from the disk to the client.    The second interesting question is that I continued to study the effect of the size of the fields on the data transfer rate. I created a collection and filled it with "for i in range(40,000,000) insert {'id':str(i),text_s:str(i)}" documents and found that on my 4.5GHz CPU Solr can process (return) only about 115 000 documents per second. It doesn't look fast - there may be a bottleneck in iterating over the set of documents returned from Solr (for example, if some expensive resource/class is created somewhere for each returned document instead of being reused)?    It seems to me that there is a benefit in such discussions and this does not concern my scenario of using Solr in any way. The equipment has become faster, its working principle has changed, taking into account the age of the Solr project, it may well turn out that some basic things should be reconsidered, rethought, changing to multithreaded or change something else. I opened the discussion itself due to the fact that my installation of SOLR worked by almost 90% equally on very old equipment and a new one is 10 times more powerful. The results achieved so far are already remarkable, but I think there is still potential for improvement. For example, a simple code review of a old-written SmileResponseWriter code made it possible to identify a bug and find a place that potentially negatively affects performance, this maybe useful for future. Best Regards,
-
To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
For additional commands, e-mail: dev-h...@solr.apache.org

RE: RE: Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-12 Thread Fikavec F

I checked how an increase in the size of the outputAggregationSize and outputBufferSize in Jetty.xml affects the data transfer speed.   - outputBufferSize - set the size of the buffer into which response content is aggregated before being sent to the client. A larger buffer can improve performance by allowing a content producer to run without blocking, however larger buffers consume more memory and may induce some latency before a client starts processing the content.   - outputAggregationSize - set the max size of the response content write that is copied into the aggregate buffer. Writes that are smaller of this size are copied into the aggregate buffer, while writes that are larger of this size will cause the aggregate buffer to be flushed and the write to be executed without being copied.Documentation: https://www.eclipse.org/jetty/javadoc/jetty-11/org/eclipse/jetty/server/HttpConfiguration.htmlThe effect of changing these parameters depends on the maximum frequency of the CPU core. The default values are optimal for 3 Ghz and allow Jetty to transfer data from the collection up to 2.9 Gb/s, an increase in parameters on such a processor has no effect.Doubling buffers on a 4.5 Ghz processor yielded the following results (wt=javabin):2.9 Gb/s - outputBufferSize=32768   outputAggregationSize=81923.6 Gb/s - outputBufferSize=65536   outputAggregationSize=163843.6 Gb/s - outputBufferSize=131072  outputAggregationSize=327684.5 Gb/s - outputBufferSize=262144  outputAggregationSize=655364.2 Gb/s - outputBufferSize=524288  outputAggregationSize=1310724.2 Gb/s - outputBufferSize=1048576 outputAggregationSize=262144   Thus, according to the documentation, an unreasonable increase in buffers will negatively affect response latency and memory consumption per connection. And according to my tests the parameter should be kept in mind only when working on processors with a frequency of more than 3 Ghz with fast wt=javabin and after testing on own data and collections. Best Regards,
-
To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
For additional commands, e-mail: dev-h...@solr.apache.org

Re: Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-09 Thread Noble Paul

@Fikavec

Can you please post the above comments on the PR? It would serve as a record

On Thu, Mar 9, 2023 at 9:07 AM Fikavec F  wrote:

> Thank you for your work Noble Paul, it's very interesting.
>You were so attentive that you noticed and replaced "else if (val
> instanceof Double) { gen.writeNumber(val.floatValue()); }" with "else if
> (val instanceof Double) { gen.writeNumber(val.doubleValue()); }". But I
> just copied the code (Line 70 -
> https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/response/SmileResponseWriter.java#L70)
>  and
> fasterxml.jackson SmileGenerator supports double (in code line 1352 -
> https://github.com/FasterXML/jackson-dataformat-smile/blob/master/src/main/java/com/fasterxml/jackson/dataformat/smile/SmileGenerator.java#L1352
>  and
> in specification
> https://github.com/FasterXML/smile-format-specification/blob/master/smile-specification.md#token-class-simple-literals-numbers)
>  It
> turns out that you found and fixed a bug in Solr SmileResponseWriter.java -
> I'm applaud, will you fix the SmileResponseWriter or report it?
>I noticed that in every JacksonJsonWriter write function call there is
> a call "WriterImpl sw = new WriterImpl(" and as a consequence, the creation
> of "JsonFactory JsonFactory = new JsonFactory();" (line 56 -
> https://github.com/apache/solr/blob/231475637e07d057d2ae1bfbe9e41e5b85b1cba5/solr/core/src/java/org/apache/solr/response/JacksonJsonWriter.java#L56)
>  but
> in the jackson documentation I read the following [Factory instances are
> thread-safe and reusable after configuration (if any). ... Factory reuse is
> important if efficiency matters; most recycling of expensive construct is
> done on per-factory basis.] (
> https://fasterxml.github.io/jackson-core/javadoc/2.13/com/fasterxml/jackson/core/JsonFactory.html)
>  As
> far as I understand, what "write" is called quite often in a query-loaded
> server, we can't avoid permanent JsonFactory creations as recommended by
> the library authors? Maybe move it to class constructor or are there
> reasons why this is bad for Solr?
>Is writeStrRaw just to check whether to use (gen.writeRawValue(val) or
> gen.writeRaw(val)) in writeStr? (I'm just already using my implementation
> and if necessary, I'll fix it in it.)
>
>The wrk benchmarking tool tests also showed some improvements. From 10%
> (with gzip) to 30% (without gzip) increase in rps when searching for random
> words in a big text field and retrieve documents. For a lot of very small
> queries with gzip enabled, the difference is not very noticeable, since the
> json output is a small percentage of the query execution time (search, data
> collection from disk and shards, gzip). But the larger the size of the
> responses received (most of all when there are few but large fields in the
> response), the more noticeable the difference, especially when the cursor
> passes through a large collection with the full data fetching. Today it
> turned out that increasing outputBufferSize and outputAggregationSize in
> jetty.xml additionally affects the speed and, it seems, allows you to
> increase it even more, I will take measurements in the near future and
> present the results.
>
> Best Regards,
>
>


-- 
-
Noble Paul

RE: Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-08 Thread Fikavec F

Thank you for your work Noble Paul, it's very interesting.   You were so attentive that you noticed and replaced "else if (val instanceof Double) { gen.writeNumber(val.floatValue()); }" with "else if (val instanceof Double) { gen.writeNumber(val.doubleValue()); }". But I just copied the code (Line 70 - https://github.com/apache/solr/blob/main/solr/core/src/java/org/apache/solr/response/SmileResponseWriter.java#L70) and fasterxml.jackson SmileGenerator supports double (in code line 1352 - https://github.com/FasterXML/jackson-dataformat-smile/blob/master/src/main/java/com/fasterxml/jackson/dataformat/smile/SmileGenerator.java#L1352 and in specification https://github.com/FasterXML/smile-format-specification/blob/master/smile-specification.md#token-class-simple-literals-numbers) It turns out that you found and fixed a bug in Solr SmileResponseWriter.java - I'm applaud, will you fix the SmileResponseWriter or report it?   I noticed that in every JacksonJsonWriter write function call there is a call "WriterImpl sw = new WriterImpl(" and as a consequence, the creation of "JsonFactory JsonFactory = new JsonFactory();" (line 56 - https://github.com/apache/solr/blob/231475637e07d057d2ae1bfbe9e41e5b85b1cba5/solr/core/src/java/org/apache/solr/response/JacksonJsonWriter.java#L56) but in the jackson documentation I read the following [Factory instances are thread-safe and reusable after configuration (if any). ... Factory reuse is important if efficiency matters; most recycling of expensive construct is done on per-factory basis.] (https://fasterxml.github.io/jackson-core/javadoc/2.13/com/fasterxml/jackson/core/JsonFactory.html) As far as I understand, what "write" is called quite often in a query-loaded server, we can't avoid permanent JsonFactory creations as recommended by the library authors? Maybe move it to class constructor or are there reasons why this is bad for Solr?   Is writeStrRaw just to check whether to use (gen.writeRawValue(val) or gen.writeRaw(val)) in writeStr? (I'm just already using my implementation and if necessary, I'll fix it in it.)    The wrk benchmarking tool tests also showed some improvements. From 10% (with gzip) to 30% (without gzip) increase in rps when searching for random words in a big text field and retrieve documents. For a lot of very small queries with gzip enabled, the difference is not very noticeable, since the json output is a small percentage of the query execution time (search, data collection from disk and shards, gzip). But the larger the size of the responses received (most of all when there are few but large fields in the response), the more noticeable the difference, especially when the cursor passes through a large collection with the full data fetching. Today it turned out that increasing outputBufferSize and outputAggregationSize in jetty.xml additionally affects the speed and, it seems, allows you to increase it even more, I will take measurements in the near future and present the results. Best Regards,

Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-08 Thread Noble Paul

I've opened the PR

https://github.com/apache/solr/pull/1440

Some tests are still failing. I'll address them later

On Tue, Mar 7, 2023 at 1:56 PM David Smiley  wrote:

> Fantastic!
> I really appreciate you working with the community on this one.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Mar 6, 2023 at 6:32 PM Fikavec F  wrote:
>
> > Thank you, you are very kind. I took measurements on two physical servers
> > with a 10 Gigabit link, the speed and time of full fetching 10 Gb
> > collection (one shard; empty "Accept-Encoding: " header; collection with
> > only id and string stored fields) are as follows:
> > original wt=json  -  419 Mb/s fetching time: 3m 26s
> > original wt=csv   -  463 Mb/s fetching time: 3m 6s
> > my wt=myfastjson  - 2.33 Gb/s fetching time:   37s
> > original wt=smile- 2.38 Gb/s fetching time:   36s
> > my wt=myfastcbor  - 2.55 Gb/s fetching time:   34s
> > original wt=javabin - 2.76 Gb/s fetching time:   31s
> >
> > With "Accept-Encoding: gzip" header:
> > original wt=json  -81  Mb/s fetching time: 12m 3s
> > my wt=myfastjson  -  114 Mb/s fetching time:8m 32s
> >
> >When I placed the test collection on 4 shards, wt=json worked at
> speeds
> > of about 430 Mb/s, and javabin, myfastcbor, myfastjson worked at speeds
> of
> > 5.2-6.7 Gb/s, however, before the start of data transfer, there was a
> > significant delay (possibly for data collection from individual shards
> and
> > sorting) and the final full fetching time did not exceed the above.
> > Downloading a 10 GB file from Jetty took place at a speed of 9.87 Gb/s.
> >The time to move the cursor through the collection documents and get
> > the field values in my python application has been reduced from 4m 24s to
> > 1m 4s, moreover, half of this minute was spent to deserialization in
> python
> > from json and cbor. No failures were noticed, everything seemed to work
> as
> > before, only very quickly (4х+ faster).
> >I dreamed of seeing Solr work at 5Gigabit+ speeds and with the help of
> > your support, everything worked out, thank you all.
> >
> > Best Regards,
> >
> >
> >
>


-- 
-
Noble Paul

Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-06 Thread David Smiley

Fantastic!
I really appreciate you working with the community on this one.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Mar 6, 2023 at 6:32 PM Fikavec F  wrote:

> Thank you, you are very kind. I took measurements on two physical servers
> with a 10 Gigabit link, the speed and time of full fetching 10 Gb
> collection (one shard; empty "Accept-Encoding: " header; collection with
> only id and string stored fields) are as follows:
> original wt=json  -  419 Mb/s fetching time: 3m 26s
> original wt=csv   -  463 Mb/s fetching time: 3m 6s
> my wt=myfastjson  - 2.33 Gb/s fetching time:   37s
> original wt=smile- 2.38 Gb/s fetching time:   36s
> my wt=myfastcbor  - 2.55 Gb/s fetching time:   34s
> original wt=javabin - 2.76 Gb/s fetching time:   31s
>
> With "Accept-Encoding: gzip" header:
> original wt=json  -81  Mb/s fetching time: 12m 3s
> my wt=myfastjson  -  114 Mb/s fetching time:8m 32s
>
>When I placed the test collection on 4 shards, wt=json worked at speeds
> of about 430 Mb/s, and javabin, myfastcbor, myfastjson worked at speeds of
> 5.2-6.7 Gb/s, however, before the start of data transfer, there was a
> significant delay (possibly for data collection from individual shards and
> sorting) and the final full fetching time did not exceed the above.
> Downloading a 10 GB file from Jetty took place at a speed of 9.87 Gb/s.
>The time to move the cursor through the collection documents and get
> the field values in my python application has been reduced from 4m 24s to
> 1m 4s, moreover, half of this minute was spent to deserialization in python
> from json and cbor. No failures were noticed, everything seemed to work as
> before, only very quickly (4х+ faster).
>I dreamed of seeing Solr work at 5Gigabit+ speeds and with the help of
> your support, everything worked out, thank you all.
>
> Best Regards,
>
>
>

RE: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-06 Thread Fikavec F

Thank you, you are very kind. I took measurements on two physical servers with a 10 Gigabit link, the speed and time of full fetching 10 Gb collection (one shard; empty "Accept-Encoding: " header; collection with only id and string stored fields) are as follows:original wt=json      -  419 Mb/s fetching time: 3m 26soriginal wt=csv       -  463 Mb/s fetching time: 3m 6smy wt=myfastjson  - 2.33 Gb/s fetching time:       37soriginal wt=smile    - 2.38 Gb/s fetching time:       36smy wt=myfastcbor  - 2.55 Gb/s fetching time:       34soriginal wt=javabin - 2.76 Gb/s fetching time:       31s With "Accept-Encoding: gzip" header:original wt=json      -    81  Mb/s fetching time: 12m 3smy wt=myfastjson  -  114 Mb/s fetching time:    8m 32s    When I placed the test collection on 4 shards, wt=json worked at speeds of about 430 Mb/s, and javabin, myfastcbor, myfastjson worked at speeds of 5.2-6.7 Gb/s, however, before the start of data transfer, there was a significant delay (possibly for data collection from individual shards and sorting) and the final full fetching time did not exceed the above. Downloading a 10 GB file from Jetty took place at a speed of 9.87 Gb/s.   The time to move the cursor through the collection documents and get the field values in my python application has been reduced from 4m 24s to 1m 4s, moreover, half of this minute was spent to deserialization in python from json and cbor. No failures were noticed, everything seemed to work as before, only very quickly (4х+ faster).   I dreamed of seeing Solr work at 5Gigabit+ speeds and with the help of your support, everything worked out, thank you all. Best Regards,

Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-06 Thread Gus Heck

Keep in mind Yoniks law of half baked patches:

"A half-baked patch with no documentation, no tests and no backwards
compatibility is better than no patch at all."

On Sun, Mar 5, 2023 at 5:43 PM Fikavec F  wrote:

> Thanks. In the coming days I will conduct testing and measurements on real
> hardware.
>Unfortunately my code is not ready to become part of the project
> directly, since this is a very serious place for changes and I am not a
> Java developer, I am not deeply familiar with the work of internal Solr
> mechanisms, the code has no tests, it does not support modes and parameters
> like the original wt=json, and I myself have a number of questions about
> code, but it would be great if someone from knowledgeable professionals
> would check my code and prepare a high-quality patch, as previously Mikhail
> Khludnev helped me here get a patch with a modified buffer. As before, I am
> happy to take part in testing such a patch, if it appears. All I did was
> replace SmileResponseWriter with JsonFactory in the source code, as I wrote
> earlier. I'm not sure that viewing my low-quality code will help
> professionals more than knowing at which part of the code there is a 4x+
> slowdown from possible speeds in order to revise and improve it.
>
>I'm prepared a repository and share the code with the changes made -
> https://github.com/Fikavec/NewAndModifiedSolrResponseWriters
> The first commit with the code of the original SmileResponseWriter so that
> it would be convenient to see what small changes I made. I placed all jar's
> from bin folders in ...
> /solr-8.11.2/server/solr-webapp/webapp/WEB-INF/lib/* and connected them via
> collection solrconfig.xml:
>
>  class="my.MyJacksonJsonResponseWriter">
>  class="my.MyJacksonCBORResponseWriter">
>
> Then I created a collection and used them as wt=myfastjson and
> wt=myfastcbor query parameters.
>
>Please let me know if there are problems in my code, especially the
> place with utf-8 raises the question, since I do not know in which encoding
> Solr transmits data to writers, Michael Gibney mentioned that in utf-16 ->
> utf-8 --> writer, in addition, there are methods writeString and
> writeRawUTF8String in jackson (
> https://fasterxml.github.io/jackson-core/javadoc/2.13/com/fasterxml/jackson/core/JsonGenerator.html)
> which one is needed after Solr passes the data to writer?
>
> Method similar to writeString(String)
> 
>  but
> that takes as its input a UTF-8 encoded String that is to be output as-is,
> without additional escaping (type of which depends on data format;
> backslashes for JSON). However, quoting that data format requires (like
> double-quotes for JSON) will be added around the value if and as necessary.
>
> Note that some backends may choose not to support this method: for
> example, if underlying destination is a Writer
> 
>  using
> this method would require UTF-8 decoding. If so, implementation may instead
> choose to throw a UnsupportedOperationException
> 
>  due
> to ineffectiveness of having to decode input.
>
> I checked my code on different utf-8 data, I didn't find any problems, but
> suddenly I used the wrong function (writeString) and there are cases when
> the data will be corrupted...
>
>Speeding up the json output would be useful to many people, but I'm not
> sure about CBOR. It turned out that CBOR is easily added (like other data
> formats from the fasterxml jackson library
> https://github.com/FasterXML/jackson#data-format-modules it is possible
> that csv, xml... will work faster with this library than the current
> implementation) as ResponseWriter, python is well supported (cbor2 fast)
> and full data fetching with cursors works 10%-20% faster than fetching data
> from Solr to python via JSON format (*this means faster in comparison with
> the modified json serializer on jackson **in python I use orjson library
> which is faster than a regular json library). I didn't find any very fast
> smile format python desereliazator, but this does not mean that many people
> needs CBOR.
>
>At the moment, everything works for me on my collections and their data
> structures and works very fast. It was surprising to me that the speed of
> regular json select with gzip has almost doubled, this could potentially
> lead to upper rps, since at full load individual server responses will
> return and end faster, I will try to check this too on real hardware using
> wrk benchmarking tool.
>
> Best Regards,
>


-- 
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-05 Thread Ishan Chattopadhyaya

Here it is, noble.
https://github.com/Fikavec/NewAndModifiedSolrResponseWriters

On Mon, 6 Mar, 2023, 7:49 am Noble Paul,  wrote:

> Hi,
>
> You can just point me to your repo and I can open a proper PR with that
> code
>
> On Mon, Mar 6, 2023, 9:43 AM Fikavec F  wrote:
>
> > Thanks. In the coming days I will conduct testing and measurements on
> real
> > hardware.
> >Unfortunately my code is not ready to become part of the project
> > directly, since this is a very serious place for changes and I am not a
> > Java developer, I am not deeply familiar with the work of internal Solr
> > mechanisms, the code has no tests, it does not support modes and
> parameters
> > like the original wt=json, and I myself have a number of questions about
> > code, but it would be great if someone from knowledgeable professionals
> > would check my code and prepare a high-quality patch, as previously
> Mikhail
> > Khludnev helped me here get a patch with a modified buffer. As before, I
> am
> > happy to take part in testing such a patch, if it appears. All I did was
> > replace SmileResponseWriter with JsonFactory in the source code, as I
> wrote
> > earlier. I'm not sure that viewing my low-quality code will help
> > professionals more than knowing at which part of the code there is a 4x+
> > slowdown from possible speeds in order to revise and improve it.
> >
> >I'm prepared a repository and share the code with the changes made -
> > https://github.com/Fikavec/NewAndModifiedSolrResponseWriters
> > The first commit with the code of the original SmileResponseWriter so
> that
> > it would be convenient to see what small changes I made. I placed all
> jar's
> > from bin folders in ...
> > /solr-8.11.2/server/solr-webapp/webapp/WEB-INF/lib/* and connected them
> via
> > collection solrconfig.xml:
> >
> >  > class="my.MyJacksonJsonResponseWriter">
> >  > class="my.MyJacksonCBORResponseWriter">
> >
> > Then I created a collection and used them as wt=myfastjson and
> > wt=myfastcbor query parameters.
> >
> >Please let me know if there are problems in my code, especially the
> > place with utf-8 raises the question, since I do not know in which
> encoding
> > Solr transmits data to writers, Michael Gibney mentioned that in utf-16
> ->
> > utf-8 --> writer, in addition, there are methods writeString and
> > writeRawUTF8String in jackson (
> >
> https://fasterxml.github.io/jackson-core/javadoc/2.13/com/fasterxml/jackson/core/JsonGenerator.html
> )
> > which one is needed after Solr passes the data to writer?
> >
> > Method similar to writeString(String)
> > <
> https://fasterxml.github.io/jackson-core/javadoc/2.13/com/fasterxml/jackson/core/JsonGenerator.html#writeString-java.lang.String->
> but
> > that takes as its input a UTF-8 encoded String that is to be output
> as-is,
> > without additional escaping (type of which depends on data format;
> > backslashes for JSON). However, quoting that data format requires (like
> > double-quotes for JSON) will be added around the value if and as
> necessary.
> >
> > Note that some backends may choose not to support this method: for
> > example, if underlying destination is a Writer
> > <
> https://docs.oracle.com/javase/8/docs/api/java/io/Writer.html?is-external=true>
> using
> > this method would require UTF-8 decoding. If so, implementation may
> instead
> > choose to throw a UnsupportedOperationException
> > <
> https://docs.oracle.com/javase/8/docs/api/java/lang/UnsupportedOperationException.html?is-external=true>
> due
> > to ineffectiveness of having to decode input.
> >
> > I checked my code on different utf-8 data, I didn't find any problems,
> but
> > suddenly I used the wrong function (writeString) and there are cases when
> > the data will be corrupted...
> >
> >Speeding up the json output would be useful to many people, but I'm
> not
> > sure about CBOR. It turned out that CBOR is easily added (like other data
> > formats from the fasterxml jackson library
> > https://github.com/FasterXML/jackson#data-format-modules it is possible
> > that csv, xml... will work faster with this library than the current
> > implementation) as ResponseWriter, python is well supported (cbor2 fast)
> > and full data fetching with cursors works 10%-20% faster than fetching
> data
> > from Solr to python via JSON format (*this means faster in comparison
> with
> > the modified json serializer on jackson **in python I use orjson library
> > which is faster than a regular json library). I didn't find any very fast
> > smile format python desereliazator, but this does not mean that many
> people
> > needs CBOR.
> >
> >At the moment, everything works for me on my collections and their
> data
> > structures and works very fast. It was surprising to me that the speed of
> > regular json select with gzip has almost doubled, this could potentially
> > lead to upper rps, since at full load individual server responses will
> > return and end faster, I will try to check this too on real hardwar

Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-05 Thread Noble Paul

Hi,

You can just point me to your repo and I can open a proper PR with that code

On Mon, Mar 6, 2023, 9:43 AM Fikavec F  wrote:

> Thanks. In the coming days I will conduct testing and measurements on real
> hardware.
>Unfortunately my code is not ready to become part of the project
> directly, since this is a very serious place for changes and I am not a
> Java developer, I am not deeply familiar with the work of internal Solr
> mechanisms, the code has no tests, it does not support modes and parameters
> like the original wt=json, and I myself have a number of questions about
> code, but it would be great if someone from knowledgeable professionals
> would check my code and prepare a high-quality patch, as previously Mikhail
> Khludnev helped me here get a patch with a modified buffer. As before, I am
> happy to take part in testing such a patch, if it appears. All I did was
> replace SmileResponseWriter with JsonFactory in the source code, as I wrote
> earlier. I'm not sure that viewing my low-quality code will help
> professionals more than knowing at which part of the code there is a 4x+
> slowdown from possible speeds in order to revise and improve it.
>
>I'm prepared a repository and share the code with the changes made -
> https://github.com/Fikavec/NewAndModifiedSolrResponseWriters
> The first commit with the code of the original SmileResponseWriter so that
> it would be convenient to see what small changes I made. I placed all jar's
> from bin folders in ...
> /solr-8.11.2/server/solr-webapp/webapp/WEB-INF/lib/* and connected them via
> collection solrconfig.xml:
>
>  class="my.MyJacksonJsonResponseWriter">
>  class="my.MyJacksonCBORResponseWriter">
>
> Then I created a collection and used them as wt=myfastjson and
> wt=myfastcbor query parameters.
>
>Please let me know if there are problems in my code, especially the
> place with utf-8 raises the question, since I do not know in which encoding
> Solr transmits data to writers, Michael Gibney mentioned that in utf-16 ->
> utf-8 --> writer, in addition, there are methods writeString and
> writeRawUTF8String in jackson (
> https://fasterxml.github.io/jackson-core/javadoc/2.13/com/fasterxml/jackson/core/JsonGenerator.html)
> which one is needed after Solr passes the data to writer?
>
> Method similar to writeString(String)
> 
>  but
> that takes as its input a UTF-8 encoded String that is to be output as-is,
> without additional escaping (type of which depends on data format;
> backslashes for JSON). However, quoting that data format requires (like
> double-quotes for JSON) will be added around the value if and as necessary.
>
> Note that some backends may choose not to support this method: for
> example, if underlying destination is a Writer
> 
>  using
> this method would require UTF-8 decoding. If so, implementation may instead
> choose to throw a UnsupportedOperationException
> 
>  due
> to ineffectiveness of having to decode input.
>
> I checked my code on different utf-8 data, I didn't find any problems, but
> suddenly I used the wrong function (writeString) and there are cases when
> the data will be corrupted...
>
>Speeding up the json output would be useful to many people, but I'm not
> sure about CBOR. It turned out that CBOR is easily added (like other data
> formats from the fasterxml jackson library
> https://github.com/FasterXML/jackson#data-format-modules it is possible
> that csv, xml... will work faster with this library than the current
> implementation) as ResponseWriter, python is well supported (cbor2 fast)
> and full data fetching with cursors works 10%-20% faster than fetching data
> from Solr to python via JSON format (*this means faster in comparison with
> the modified json serializer on jackson **in python I use orjson library
> which is faster than a regular json library). I didn't find any very fast
> smile format python desereliazator, but this does not mean that many people
> needs CBOR.
>
>At the moment, everything works for me on my collections and their data
> structures and works very fast. It was surprising to me that the speed of
> regular json select with gzip has almost doubled, this could potentially
> lead to upper rps, since at full load individual server responses will
> return and end faster, I will try to check this too on real hardware using
> wrk benchmarking tool.
>
> Best Regards,
>

RE: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-05 Thread Fikavec F

Thanks. In the coming days I will conduct testing and measurements on real hardware.   Unfortunately my code is not ready to become part of the project directly, since this is a very serious place for changes and I am not a Java developer, I am not deeply familiar with the work of internal Solr mechanisms, the code has no tests, it does not support modes and parameters like the original wt=json, and I myself have a number of questions about code, but it would be great if someone from knowledgeable professionals would check my code and prepare a high-quality patch, as previously Mikhail Khludnev helped me here get a patch with a modified buffer. As before, I am happy to take part in testing such a patch, if it appears. All I did was replace SmileResponseWriter with JsonFactory in the source code, as I wrote earlier. I'm not sure that viewing my low-quality code will help professionals more than knowing at which part of the code there is a 4x+ slowdown from possible speeds in order to revise and improve it.    I'm prepared a repository and share the code with the changes made - https://github.com/Fikavec/NewAndModifiedSolrResponseWritersThe first commit with the code of the original SmileResponseWriter so that it would be convenient to see what small changes I made. I placed all jar's from bin folders in ... /solr-8.11.2/server/solr-webapp/webapp/WEB-INF/lib/* and connected them via collection solrconfig.xml:    Then I created a collection and used them as wt=myfastjson and wt=myfastcbor query parameters.    Please let me know if there are problems in my code, especially the place with utf-8 raises the question, since I do not know in which encoding Solr transmits data to writers, Michael Gibney mentioned that in utf-16 -> utf-8 --> writer, in addition, there are methods writeString and writeRawUTF8String in jackson (https://fasterxml.github.io/jackson-core/javadoc/2.13/com/fasterxml/jackson/core/JsonGenerator.html) which one is needed after Solr passes the data to writer?Method similar to writeString(String) but that takes as its input a UTF-8 encoded String that is to be output as-is, without additional escaping (type of which depends on data format; backslashes for JSON). However, quoting that data format requires (like double-quotes for JSON) will be added around the value if and as necessary.Note that some backends may choose not to support this method: for example, if underlying destination is a Writer using this method would require UTF-8 decoding. If so, implementation may instead choose to throw a UnsupportedOperationException due to ineffectiveness of having to decode input.I checked my code on different utf-8 data, I didn't find any problems, but suddenly I used the wrong function (writeString) and there are cases when the data will be corrupted...    Speeding up the json output would be useful to many people, but I'm not sure about CBOR. It turned out that CBOR is easily added (like other data formats from the fasterxml jackson library https://github.com/FasterXML/jackson#data-format-modules it is possible that csv, xml... will work faster with this library than the current implementation) as ResponseWriter, python is well supported (cbor2 fast) and full data fetching with cursors works 10%-20% faster than fetching data from Solr to python via JSON format (*this means faster in comparison with the modified json serializer on jackson **in python I use orjson library which is faster than a regular json library). I didn't find any very fast smile format python desereliazator, but this does not mean that many people needs CBOR.   At the moment, everything works for me on my collections and their data structures and works very fast. It was surprising to me that the speed of regular json select with gzip has almost doubled, this could potentially lead to upper rps, since at full load individual server responses will return and end faster, I will try to check this too on real hardware using wrk benchmarking tool. Best Regards,

Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-05 Thread Ishan Chattopadhyaya

Wow, that's absolutely fantastic. Would you like to contribute it upstream,
maybe a PR?

On Sun, 5 Mar, 2023, 9:21 pm Fikavec F,  wrote:

> Thanks to everyone for your help (especially Mikhail Khludnev and Michael
> Gibney)!
>
>I rewrote JSONResponseWriter using fasterxml jackson library
> (com.fasterxml.jackson.core.JsonGenerator, I took the SmileResponseWriter
> code as a basis) and added it to the collection as my own
> queryResponseWriter in solrconfig.xml - in the test virtual infrastructure
> I got 4x+ acceleration in json format without gzip (set http header
> 'Accept-Encoding: '), and about 2x with gzip enabled (with set http header
> 'Accept-Encoding: gzip'). I think that on real hardware, the acceleration
> may be a little more. CBOR from the jackson library turned out to be also
> very fast and easy to add, it is also supported in python by the cbor2
> library and works very very fast, which is not to say about smile.
>
>Now we know for sure that Solr is capable of sending data in json
> format at speeds of more than a few gigabits+, but the default
> JSONResponseWriter cannot overcome 450 megabits - using the jackson library
> for serialization solves this problem (
> https://github.com/wizzardo/json-benchmarks).
>
> Best Regards,
> - To
> unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional
> commands, e-mail: dev-h...@solr.apache.org

RE: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-03-05 Thread Fikavec F

Thanks to everyone for your help (especially Mikhail Khludnev and Michael Gibney)!    I rewrote JSONResponseWriter using fasterxml jackson library (com.fasterxml.jackson.core.JsonGenerator, I took the SmileResponseWriter code as a basis) and added it to the collection as my own queryResponseWriter in solrconfig.xml - in the test virtual infrastructure I got 4x+ acceleration in json format without gzip (set http header 'Accept-Encoding: '), and about 2x with gzip enabled (with set http header 'Accept-Encoding: gzip'). I think that on real hardware, the acceleration may be a little more. CBOR from the jackson library turned out to be also very fast and easy to add, it is also supported in python by the cbor2 library and works very very fast, which is not to say about smile.    Now we know for sure that Solr is capable of sending data in json format at speeds of more than a few gigabits+, but the default JSONResponseWriter cannot overcome 450 megabits - using the jackson library for serialization solves this problem (https://github.com/wizzardo/json-benchmarks). Best Regards,
-
To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
For additional commands, e-mail: dev-h...@solr.apache.org

Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-27 Thread Ishan Chattopadhyaya

> Does Solr have a project to track basic performance metrics from version
to version, similar to Rally (the macro-benchmarking framework for
Elasticsearch)

?

There's an ongoing effort to have periodic benchmarks. This type of testing
would be ideal for repeating periodically, hence I asked.
JIRA: https://issues.apache.org/jira/browse/SOLR-16525
Current link: http://mostly.cool/graph.html (click branch_9x on left).

On Tue, Feb 28, 2023 at 6:29 AM Fikavec F  wrote:

> Thank you for helping me figure out the performance of "Response Writers".
> If you wish, you can close the topic or move it to issues under the title
> performance of non-binary types of "Response Writers" (if in your opinion
> the difference of 4-8 times between wt=javabin and wt=json is not normal).
> Maybe it is worth adding to the documentation a mention of the order of the
> difference in the speeds of "Response Writers" in /select and /export
> handlers? I would really like wt=json to work as fast as possible, as it is
> popular and compatible, moreover, Solr itself works much faster than
> wt=json.
>
> I have read and tested the specified wt=smile format (it is slower than
> java bin, but also significantly faster than wt=json). Does the JDBC driver
> (when working from python) offered in Solr 9.1.1 work internally using
> javabin for client-server interaction (that is, data is received on it at
> javabin speeds or as in wt=json)?
>
> As I wrote earlier - jetty is not a bottleneck in this issue and in nginx
> activated { io threads; sendfile on; } as in the article on configuring
> nginx . I
> haven't thoroughly tested jetty specifically, just quickly checked that
> jetty is able to transfer data (files) faster than when they are fetched
> from Solr. I saw that jetty can work to transmit data at very high speeds
> of the order of 6-9 gigabytes, and this was enough for me, I did not think
> to compare and demand from it the speed of work as nginx optimized for file
> transfer.
>
> Michael Gibney - without using sorting in my tests on Solr 9.1.1, fetching
> looks a bit faster, but not so much that it is significant:
>
>- 409 Mb/s sort=id asc  wt=javabin
>- 411 Mb/s without sort wt=javabin
>- 1.40 Gb/s sort=id asc  wt=javabin
>- 1.48 Gb/s without sort wt=javabin
>- 2.38 Gb/s sort=id asc  wt=smile
>- 2.59 Gb/s without sort wt=smile
>
> All measurement results:
>
> - SOLR 9.1.1 tests 28.02.2023 field with stored=true
> docValues=false -->
> !!![sort=id asc]
> /select HANDLER wt=json sort=id asc
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=json&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.03G in 3m 30s
> 2023-02-27 21:55:53 (409 Mb/s) - ‘/dev/null’ saved [10772687921]
>
> /select HANDLER wt=javabin sort=id asc
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=javabin&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.01G in 61s
> 2023-02-27 22:01:32 (1.40 Gb/s) - ‘/dev/null’ saved [10749142324]
>
> /select HANDLER wt=smile sort=id asc
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=smile&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.01G in 36s
> 2023-02-27 22:31:40 (2.38 Gb/s) - ‘/dev/null’ saved [10748488299]
>
> !!![without sort]
> /select HANDLER wt=json without sort
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=json&q=*%3A*&rows=100'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.03G in 3m 30s
> 2023-02-27 22:06:31 (411 Mb/s) - ‘/dev/null’ saved [10772687903]
>
> /select HANDLER wt=javabin without sort
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=javabin&q=*%3A*&rows=100'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.01G in 58s
> 2023-02-27 22:13:36 (1.48 Gb/s) - ‘/dev/null’ saved [10749142310]
>
> /select HANDLER wt=smile without sort
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=smile&q=*%3A*&rows=100'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.01G in 33s
> 2023-02-27 22:29:38 (2.59 Gb/s) - ‘/dev/null’ saved [10748488286]
>
>
> Ishan Chattopadhyaya - yes, regarding my tests, it looks like javabean in
> Solr 9.1.1 significantly degraded in speed relative to version 8.11.2.
> Since I was taking measurements in a 10 Gigabit network infrastructure
> (emulated on Vms and real) with special opti

RE: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-27 Thread Fikavec F

Thank you for helping me figure out the performance of "Response Writers". If you wish, you can close the topic or move it to issues under the title performance of non-binary types of "Response Writers" (if in your opinion the difference of 4-8 times between wt=javabin and wt=json is not normal). Maybe it is worth adding to the documentation a mention of the order of the difference in the speeds of "Response Writers" in /select and /export handlers? I would really like wt=json to work as fast as possible, as it is popular and compatible, moreover, Solr itself works much faster than wt=json. I have read and tested the specified wt=smile format (it is slower than java bin, but also significantly faster than wt=json). Does the JDBC driver (when working from python) offered in Solr 9.1.1 work internally using javabin for client-server interaction (that is, data is received on it at javabin speeds or as in wt=json)? As I wrote earlier - jetty is not a bottleneck in this issue and in nginx activated { io threads; sendfile on; } as in the article on configuring nginx. I haven't thoroughly tested jetty specifically, just quickly checked that jetty is able to transfer data (files) faster than when they are fetched from Solr. I saw that jetty can work to transmit data at very high speeds of the order of 6-9 gigabytes, and this was enough for me, I did not think to compare and demand from it the speed of work as nginx optimized for file transfer. Michael Gibney - without using sorting in my tests on Solr 9.1.1, fetching looks a bit faster, but not so much that it is significant:409 Mb/s sort=id asc  wt=javabin411 Mb/s without sort wt=javabin1.40 Gb/s sort=id asc  wt=javabin1.48 Gb/s without sort wt=javabin2.38 Gb/s sort=id asc  wt=smile2.59 Gb/s without sort wt=smileAll measurement results:- SOLR 9.1.1 tests 28.02.2023 field with stored=true docValues=false -->!!![sort=id asc]/select HANDLER wt=json sort=id ascwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=json&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.03G in 3m 30s2023-02-27 21:55:53 (409 Mb/s) - ‘/dev/null’ saved [10772687921] /select HANDLER wt=javabin sort=id ascwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=javabin&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 61s2023-02-27 22:01:32 (1.40 Gb/s) - ‘/dev/null’ saved [10749142324] /select HANDLER wt=smile sort=id ascwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=smile&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 36s2023-02-27 22:31:40 (2.38 Gb/s) - ‘/dev/null’ saved [10748488299] !!![without sort]/select HANDLER wt=json without sortwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=json&q=*%3A*&rows=100' http://192.168.220.135:8983/solr/test_collection/select10.03G in 3m 30s2023-02-27 22:06:31 (411 Mb/s) - ‘/dev/null’ saved [10772687903] /select HANDLER wt=javabin without sortwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=javabin&q=*%3A*&rows=100' http://192.168.220.135:8983/solr/test_collection/select10.01G in 58s2023-02-27 22:13:36 (1.48 Gb/s) - ‘/dev/null’ saved [10749142310] /select HANDLER wt=smile without sortwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=smile&q=*%3A*&rows=100' http://192.168.220.135:8983/solr/test_collection/select10.01G in 33s2023-02-27 22:29:38 (2.59 Gb/s) - ‘/dev/null’ saved [10748488286] Ishan Chattopadhyaya - yes, regarding my tests, it looks like javabean in Solr 9.1.1 significantly degraded in speed relative to version 8.11.2. Since I was taking measurements in a 10 Gigabit network infrastructure (emulated on Vms and real) with special optimization of the OS for it (enable Jumbo Frames mtu 9000; tcp_window_scaling; 128MB tcp_wmem and tcp_rmem) and disabled gzip, then the difference in performance could become noticeable, which under normal conditions was not noticeable at all due to the impossibility of achieving such network speeds with standard settings. Therefore, my aspirations and measurements at speeds over gigabit are not yet a mass need and phenomenon and may not be worth close attention. The most interesting thing is that in today's measurement for Michael Gibney in Solr 9.1.1 (presented above) javabin was slower than wt=smile, but in version 8.11.2 wt=smile was fast, but noticeably slower than javabin. If necessary, I will provide you with everything you may need to reproduce. Performance is a very complicated matter, it is affected by a lot of things, it is possible that the slowdown occurred du

Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-27 Thread Michael Gibney

This is all really interesting, thanks for the detailed feedback (esp.
for trying out 9.1)! I think the optimization I mentioned from 9.1 is
not relevant, but I should have mentioned: the optimization in 9.1
only applies to `*:*` with sort-by-score. Including the
`sort=[anything-other-than-score]` param means sort is relevant, so
you can't take the optimization (which only applies when it's possible
to avoid sorting altogether) -- and sorting top-N for such a large
number of rows is definitely going to take some time.

I'm curious to know whether you see a difference on 9.1.1 if you omit
the `sort` param (defaulting to sort-by-score, which should allow the
sort-bypass optimization to be taken). But again, it doesn't look like
this accounts for the increased latency you're observing (in fact I'm
mainly curious because this might help narrow down where to look for
possible cause of the reported performance degradation in 9.1.1 --
definitely worth looking into).

"string" values for javabin are basically written direct bytes to an
OutputStream, whereas for the text response formats, string values'
lucene-native utf8 representation are converted to utf16 and then
written back as text (which is then of course converted into utf8 in
the context of the response). I suspect that if you're really chasing
top performance, it'd be well worth finding (or even just
implementing!) a client that "speaks" javabin.

Like Kevin, I'm also surprised by the difference between jetty and
Nginx serving static files. I'm curious, does your Nginx config have
`sendfile` directive enabled? ("By default, NGINX handles file
transmission itself and copies the file into the buffer before sending
it. Enabling the sendfile directive eliminates the step of copying the
data into the buffer and enables direct copying data from one file
descriptor to another.")

One thought: to further isolate "core" solr concerns from
serialization/network concerns, it could be useful to separate latency
according to before/after first bytes are received.

On Sun, Feb 26, 2023 at 8:11 PM Fikavec F  wrote:
>
> David Smiley, sorry for my terminology, I’m used to calling a full data 
> fetching by small parts from DB table (collection) as "scrolling". Of course, 
> in Solr cursors (cursorMark) are designed for this and I use them. Large 
> "rows" values in my examples (measurements) are needed to show the speed at 
> which data from a 10 GB test collection is transmitted from Solr. When using 
> cursors and passing through a 250+ gigabyte collection, data is transmitted 
> at the same speed as with a single call /select, but with delays between 
> scrolls. In practice, I noticed that regardless of the performance of the 
> hardware (SAS disk vs RAMdisk; 1 Gbps vs 10 Gbps network; CPU with a core 
> frequency of 30% more), the full data fetching time does not change and data 
> transfer speed keep around 350 Megabits. At the same time, if instead of 
> select from Solr, start downloading the file via Solr Jetty from the folder 
> ".../solr-webapp/webapp/test.bin" the speed quickly goes beyond the gigabit. 
> And here it doesn't matter if we take top-X and how big this X is - it still 
> happens 4-8x slower than the capabilities of Jetty and Solr and that's not 
> good. It looks like the slowdown affects everything "responce writes" (json, 
> xml, csv, python) I've tested except javabin, which just demonstrates that a 
> bottleneck is possible not deep in Solr, but somewhere at the level of 
> transformation-data transfer by the "responce writers" (except javabin). I 
> don't know where to look for the problem further than the FastWriter output 
> buffer, but I hope the specialists will succeed.
>
> To the remark of Michael Gibney, I tested the speed of Solr 9.1.1. Strangely, 
> it turned out to be much slower than Solr 8.11.2, even if use /export, 
> DocValues and javabin responce writer:
>
> 1.50 Gb/s - HANDLER /select;  wt=javabin; with stored=true docValues=false 
> field
> 489 Mb/s  - HANDLER /select;  wt=csv;   with stored=true docValues=false 
> field
> 459 Mb/s  - HANDLER /select;  wt=json;   with stored=true docValues=false 
> field
> 433 Mb/s  - HANDLER /export;  wt=javabin; works only on docValues=true feild
> 194 Mb/s  - HANDLER /select;  wt=json;with stored=false docValues=true 
> field
>
> All conditions are the same as before, just Solr 9.1.1 is installed on the 
> ram disk (java --version - OpenJDK 64-Bit Server VM (build 
> 11.0.17+8-post-Ubuntu-1ubuntu220.04; OS - Ubuntu 20.04.3 LTS; 
> SOLR_JAVA_MEM="-Xms8g -Xmx8g" running in cloud mode, other - defaults). It is 
> not so difficult to create ram disk and try to repeat the above commands on 
> 127.0.0.1 or at least in the gigabit network to see how far the speed from 
> gigabit in the absence of a bottleneck in the disk or network.
>
> All measurement results:
>
>
> - SOLR 9.1.1 tests -
> ---[ /select HANDLER ] ---
> /select HANDLER wt=json
> wget --report-speed=bits

Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-27 Thread Ishan Chattopadhyaya

Fikavec, thanks for your thorough analysis. It will help Solr.

To be clear, 8.11 vs 9.1 Solr has a degradation from 3.66gbps to 1.5gbps
with wt=javabin? If yes, this requires fixing this. Can you please help us
reproduce it?

On Mon, 27 Feb, 2023, 6:41 am Fikavec F,  wrote:

> David Smiley, sorry for my terminology, I’m used to calling a full data
> fetching by small parts from DB table (collection) as "scrolling". Of
> course, in Solr cursors (cursorMark) are designed for this and I use them.
> Large "rows" values in my examples (measurements) are needed to show the
> speed at which data from a 10 GB test collection is transmitted from Solr.
> When using cursors and passing through a 250+ gigabyte collection, data is
> transmitted at the same speed as with a single call /select, but with
> delays between scrolls. In practice, I noticed that regardless of the
> performance of the hardware (SAS disk vs RAMdisk; 1 Gbps vs 10 Gbps
> network; CPU with a core frequency of 30% more), the full data fetching
> time does not change and data transfer speed keep around 350 Megabits. At
> the same time, if instead of select from Solr, start downloading the file
> via Solr Jetty from the folder ".../solr-webapp/webapp/test.bin" the speed
> quickly goes beyond the gigabit. And here it doesn't matter if we take
> top-X and how big this X is - it still happens 4-8x slower than the
> capabilities of Jetty and Solr and that's not good. It looks like the
> slowdown affects everything "responce writes" (json, xml, csv, python) I've
> tested except javabin, which just demonstrates that a bottleneck is
> possible not deep in Solr, but somewhere at the level of
> transformation-data transfer by the "responce writers" (except javabin). I
> don't know where to look for the problem further than the FastWriter output
> buffer, but I hope the specialists will succeed.
>
> To the remark of Michael Gibney, I tested the speed of Solr 9.1.1.
> Strangely, it turned out to be much slower than Solr 8.11.2, even if use
> /export, DocValues and javabin responce writer:
>
>- 1.50 Gb/s - HANDLER /select;  wt=javabin; with stored=true
>docValues=false field
>- 489 Mb/s  - HANDLER /select;  wt=csv;   with stored=true
>docValues=false field
>- 459 Mb/s  - HANDLER /select;  wt=json;   with stored=true
>docValues=false field
>- 433 Mb/s  - HANDLER /export;  wt=javabin; works only on
>docValues=true feild
>- 194 Mb/s  - HANDLER /select;  wt=json;with stored=false
>docValues=true field
>
> All conditions are the same as before, just Solr 9.1.1 is installed on the
> ram disk (java --version - OpenJDK 64-Bit Server VM (build
> 11.0.17+8-post-Ubuntu-1ubuntu220.04; OS - Ubuntu 20.04.3 LTS;
> SOLR_JAVA_MEM="-Xms8g -Xmx8g" running in cloud mode, other - defaults). It
> is not so difficult to create ram disk and try to repeat the above commands
> on 127.0.0.1 or at least in the gigabit network to see how far the speed
> from gigabit in the absence of a bottleneck in the disk or network.
>
> All measurement results:
>
>
> - SOLR 9.1.1 tests -
> ---[ /select HANDLER ] ---
> /select HANDLER wt=json
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=json&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.03G in 3m 8s
> 2023-02-26 19:09:45 (459 Mb/s) - ‘/dev/null’ saved [10772687921]
>
> /select HANDLER wt=javabin
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=javabin&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.01G in 57s
> 2023-02-26 19:26:55 (1.50 Gb/s) - ‘/dev/null’ saved [10749142324]
>
> /select HANDLER wt=csv
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=csv&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.01G in 2m 56s
> 2023-02-26 19:30:06 (489 Mb/s) - ‘/dev/null’ saved [10751204971]
>
> # 2. Experiments with docValues=true stored=false (for testing /export
> HANDLER)
> ---[ /select HANDLER ] ---
> /select HANDLER wt=json
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=json&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.03G in 7m 24s
> 2023-02-26 20:35:55 (194 Mb/s) - ‘/dev/null’ saved [10772687921]
>
> /select HANDLER wt=javabin
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=javabin&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.01G in 4m 13s
> 2023-02-26 20:45:00 (340 Mb/s) - ‘/dev/null’ saved [10749142324]
>
> /select HANDLER wt=csv
> wget --report-speed=bits --server-response -

RE: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-26 Thread Fikavec F

David Smiley, sorry for my terminology, I’m used to calling a full data fetching by small parts from DB table (collection) as "scrolling". Of course, in Solr cursors (cursorMark) are designed for this and I use them. Large "rows" values in my examples (measurements) are needed to show the speed at which data from a 10 GB test collection is transmitted from Solr. When using cursors and passing through a 250+ gigabyte collection, data is transmitted at the same speed as with a single call /select, but with delays between scrolls. In practice, I noticed that regardless of the performance of the hardware (SAS disk vs RAMdisk; 1 Gbps vs 10 Gbps network; CPU with a core frequency of 30% more), the full data fetching time does not change and data transfer speed keep around 350 Megabits. At the same time, if instead of select from Solr, start downloading the file via Solr Jetty from the folder ".../solr-webapp/webapp/test.bin" the speed quickly goes beyond the gigabit. And here it doesn't matter if we take top-X and how big this X is - it still happens 4-8x slower than the capabilities of Jetty and Solr and that's not good. It looks like the slowdown affects everything "responce writes" (json, xml, csv, python) I've tested except javabin, which just demonstrates that a bottleneck is possible not deep in Solr, but somewhere at the level of transformation-data transfer by the "responce writers" (except javabin). I don't know where to look for the problem further than the FastWriter output buffer, but I hope the specialists will succeed. To the remark of Michael Gibney, I tested the speed of Solr 9.1.1. Strangely, it turned out to be much slower than Solr 8.11.2, even if use /export, DocValues and javabin responce writer:1.50 Gb/s - HANDLER /select;  wt=javabin; with stored=true docValues=false field489 Mb/s  - HANDLER /select;  wt=csv;       with stored=true docValues=false field459 Mb/s  - HANDLER /select;  wt=json;       with stored=true docValues=false field433 Mb/s  - HANDLER /export;  wt=javabin; works only on docValues=true feild194 Mb/s  - HANDLER /select;  wt=json;    with stored=false docValues=true fieldAll conditions are the same as before, just Solr 9.1.1 is installed on the ram disk (java --version - OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu220.04; OS - Ubuntu 20.04.3 LTS; SOLR_JAVA_MEM="-Xms8g -Xmx8g" running in cloud mode, other - defaults). It is not so difficult to create ram disk and try to repeat the above commands on 127.0.0.1 or at least in the gigabit network to see how far the speed from gigabit in the absence of a bottleneck in the disk or network. All measurement results: - SOLR 9.1.1 tests ----[ /select HANDLER ] ---/select HANDLER wt=jsonwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=json&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.03G in 3m 8s2023-02-26 19:09:45 (459 Mb/s) - ‘/dev/null’ saved [10772687921] /select HANDLER wt=javabinwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=javabin&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 57s2023-02-26 19:26:55 (1.50 Gb/s) - ‘/dev/null’ saved [10749142324] /select HANDLER wt=csvwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=csv&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 2m 56s2023-02-26 19:30:06 (489 Mb/s) - ‘/dev/null’ saved [10751204971] # 2. Experiments with docValues=true stored=false (for testing /export HANDLER)---[ /select HANDLER ] ---/select HANDLER wt=jsonwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=json&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.03G in 7m 24s2023-02-26 20:35:55 (194 Mb/s) - ‘/dev/null’ saved [10772687921] /select HANDLER wt=javabinwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=javabin&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 4m 13s2023-02-26 20:45:00 (340 Mb/s) - ‘/dev/null’ saved [10749142324] /select HANDLER wt=csvwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=csv&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 7m 5s2023-02-26 21:25:35 (202 Mb/s) - ‘/dev/null’ saved [10751204971] ---[ /export HANDLER ] ---/export HANDLER wt=jsonwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' "http://192.168.220.135:8983/solr/test_collection/export?indent=false&wt=json&q=*%3A*&sort=id%20desc&fl=id,text_sn"10.01G in 4m 7s2023-02-26 21:32:40 (349 M

Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-26 Thread Mikhail Khludnev

>
> Isn't there a deserializer of Solr javabin format in python/json

Well, you can try to marry
https://solr.apache.org/guide/solr/latest/query-guide/response-writers.html#smile-response-writer
 + https://github.com/jhosmer/PySmile

On Mon, Feb 27, 2023 at 12:46 AM Fikavec F  wrote:

> Thank you for your help with slow single threaded data receiving from
> Solr. Today I was able to reach a speed of 3Gigabit+ and got results that
> may be useful in the future.
>
> I turned out to be wrong in assuming that the main problem is in the
> FastWriter output buffer, but this was the most obvious thing I found while
> studying the source code of different Solr Response Writers (they are
> inherited from him). I can't determine exactly where the problem is, but
> according to my experiments, it doesn't seem to be too deep, but somewhere
> in the process of the final transformation-sending data to the user. I
> would think that the problem is in the library/code that converts data to
> the output format, but it is very strange that the speed is almost
> independent (not by orders of magnitude, rather not significantly) on the
> selected format (csv, json, xml, python...), whether it is the fastest csv
> or the slowest xml. Perhaps the data is slow to arrive at these functions
> or is slowly converted/deserialized to them from the internal format, but
> the difference is up to 8 times slower than it could be.
>
> Just in case, I checked the performance of Streaming Expressions
> 
> (the speed is less than with /select, and a very long wait for a response
> from the server before the data transfer begins and a huge memory
> consumption) and Exporting Result Sets
> 
> too, but Exporting Result Sets only works with DocValues
> 
> fields, and are not suitable for general use, since multiple restrictions
> 
> are imposed on DocValues fields (for example, you cannot use DocValues on
> solr.TextField fields, only StrField).
>
> Here's what I got:
>
>- 3.66 Gb/s - HANDLER /export; wt=javabin; works only on
>docValues=true feild
>- *2.95 Gb/s* - *HANDLER /select;  wt=javabin; with stored=true
>docValues=false field*
>- 1.46 Gb/s - HANDLER /export; wt=json; works only on docValues=true
>feild
>- 455 Mb/s  - HANDLER /select;  wt=csv; with stored=true
>docValues=false field
>- *361 Mb/s*  *- HANDLER /select;  wt=json; with stored=true
>docValues=false field*
>
> In any case, the difference between javabin and other Response Writers is
> enormous. This gives me hope that the problem is not deep internal and can
> be fixed in the future. Solr itself can work at 3Gigabit speeds and this is
> not the limit of its Jetty, i.e. there is a lot to improve.
> Isn't there a deserializer of Solr javabin format in python/json - while
> wt=json is so slow, on the client, the data could be quickly received as
> javabin and converted into what was needed as MongoDB PyMongo client does
> with BSON (Binary JSON) format
> ?
>
> All measurement results:
>
>
> # 1. Experiments with stored=true docValues=false
> ---[ /select HANDLER ] ---
> /select HANDLER wt=json
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=json&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.03G in *3m 59s*
> 2023-02-26 15:05:45 (*361 Mb/s*) - ‘/dev/null’ saved [10772687921]
>
> /select HANDLER wt=javabin
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=javabin&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.01G in *29s*
> 2023-02-26 15:08:59 (*2.95 Gb/s*) - ‘/dev/null’ saved [10749142324]
>
> /select HANDLER wt=csv
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=csv&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.01G in *3m 9s*
> 2023-02-26 15:14:24 (*455 Mb/s*) - ‘/dev/null’ saved [10751204971]
>
> # 2. Experiments with docValues=true stored=false (for testing /export
> HANDLER)
> ---[ /select HANDLER ] ---
> /select HANDLER wt=json
> wget --report-speed=bits --server-response -O /dev/null
> --header='Accept-Encoding: ' --post-data
> 'indent=false&wt=json&q=*%3A*&rows=100&sort=id%20asc'
> http://192.168.220.135:8983/solr/test_collection/select
> 10.03G in *4m 2s*
> 2023-02-26 14:24:21 (*357 Mb/s*) - ‘/dev/null’ saved [10772687921]
>
> /select HANDLER wt=jav

RE: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-26 Thread Fikavec F

Thank you for your help with slow single threaded data receiving from Solr. Today I was able to reach a speed of 3Gigabit+ and got results that may be useful in the future. I turned out to be wrong in assuming that the main problem is in the FastWriter output buffer, but this was the most obvious thing I found while studying the source code of different Solr Response Writers (they are inherited from him). I can't determine exactly where the problem is, but according to my experiments, it doesn't seem to be too deep, but somewhere in the process of the final transformation-sending data to the user. I would think that the problem is in the library/code that converts data to the output format, but it is very strange that the speed is almost independent (not by orders of magnitude, rather not significantly) on the selected format (csv, json, xml, python...), whether it is the fastest csv or the slowest xml. Perhaps the data is slow to arrive at these functions or is slowly converted/deserialized to them from the internal format, but the difference is up to 8 times slower than it could be. Just in case, I checked the performance of Streaming Expressions (the speed is less than with /select, and a very long wait for a response from the server before the data transfer begins and a huge memory consumption) and Exporting Result Sets too, but Exporting Result Sets only works with DocValues fields, and are not suitable for general use, since multiple restrictions are imposed on DocValues fields (for example, you cannot use DocValues on solr.TextField fields, only StrField). Here's what I got:3.66 Gb/s - HANDLER /export; wt=javabin; works only on docValues=true feild2.95 Gb/s - HANDLER /select;  wt=javabin; with stored=true docValues=false field1.46 Gb/s - HANDLER /export; wt=json; works only on docValues=true feild455 Mb/s  - HANDLER /select;  wt=csv; with stored=true docValues=false field361 Mb/s  - HANDLER /select;  wt=json; with stored=true docValues=false fieldIn any case, the difference between javabin and other Response Writers is enormous. This gives me hope that the problem is not deep internal and can be fixed in the future. Solr itself can work at 3Gigabit speeds and this is not the limit of its Jetty, i.e. there is a lot to improve.Isn't there a deserializer of Solr javabin format in python/json - while wt=json is so slow, on the client, the data could be quickly received as javabin and converted into what was needed as MongoDB PyMongo client does with BSON (Binary JSON) format? All measurement results: # 1. Experiments with stored=true docValues=false---[ /select HANDLER ] ---/select HANDLER wt=jsonwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=json&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.03G in 3m 59s2023-02-26 15:05:45 (361 Mb/s) - ‘/dev/null’ saved [10772687921] /select HANDLER wt=javabinwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=javabin&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 29s2023-02-26 15:08:59 (2.95 Gb/s) - ‘/dev/null’ saved [10749142324] /select HANDLER wt=csvwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=csv&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 3m 9s2023-02-26 15:14:24 (455 Mb/s) - ‘/dev/null’ saved [10751204971]                # 2. Experiments with docValues=true stored=false (for testing /export HANDLER)---[ /select HANDLER ] ---/select HANDLER wt=jsonwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=json&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.03G in 4m 2s2023-02-26 14:24:21 (357 Mb/s) - ‘/dev/null’ saved [10772687921] /select HANDLER wt=javabinwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=javabin&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 71s2023-02-26 14:27:15 (1.21 Gb/s) - ‘/dev/null’ saved [10749142324] /select HANDLER wt=csvwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' --post-data 'indent=false&wt=csv&q=*%3A*&rows=100&sort=id%20asc' http://192.168.220.135:8983/solr/test_collection/select10.01G in 3m 20s2023-02-26 14:32:39 (430 Mb/s) - ‘/dev/null’ saved [10751204971] ---[ /export HANDLER ] ---/export HANDLER wt=jsonwget --report-speed=bits --server-response -O /dev/null --header='Accept-Encoding: ' "http://192.168.220.135:8983/solr/test_collection/export?indent=false&wt=json&q=*%3A*&sort=id%20desc&fl=id,text_sn"10.01G in 59s2023-02-26 14:34:35 (1.46 Gb/s) - ‘/dev/null’ saved [10751758398] /export HANDLER wt=javabinwget --report-

Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-26 Thread Michael Gibney

> I'm not sure there's a shortcut bypassing ordering results through heap

To expand on this a bit: the behavior Mikhail describes changes as of
solr 9.1 (https://issues.apache.org/jira/browse/SOLR-14765), which
introduces exactly the proposed bypass. The extra overhead (pre-9.1)
scales linear wrt overall number of docs in the overall result set (in
the *:* case, the number of docs in the index). This could be a
contributing factor to the latency you're observing; unfortunately,
short of upgrading to 9.1 (or backporting the related patch?), I'm not
sure there's any other workaround (because of how deep in the code the
result set sorting is).

The `sort=_docid_ asc` hack is interesting ... tbh I'm surprised this
makes a difference vs constant-score, because having been pretty deep
in the sorting code, there's no special-casing of that case, and I'd
think it should behave similar to constant score. I suspect this would
make a difference relative to main queries that are _not_
constant-score, for cases where you don't care about scoring -- but
then I'd expect equivalent behavior to be achievable by wrapping the
main query as a constant score (e.g., `q=(some query)^=1.0`).

I'm curious (and don't know or don't recall) how this all plays out
with the export handler -- I'll take a deeper look at that when I get
a chance. (+1 to export handler being much better for the "bulk
export" use case).

On Sun, Feb 26, 2023 at 2:07 AM Mikhail Khludnev  wrote:
>
> As being said above, the speed of streaming file to socket, via OS
> internals is hardly achievable with java code crunching bytes through heap.
> Using RamDirectory might push on GC ,it's rather better to stick with the
> default one and leave enough RAM for file cache.
> Regarding  the actual params: q=*%3A*&rows=10 I'm not sure there's a
> shortcut bypassing ordering results through heap (even *:* score is
> constant), quite often sort=_docid_ asc allows to avoid sorting, try it.
> Also, there is https://solr.apache.org/guide/8_6/exporting-result-sets.html
> which might be better suites for straightforward download, don't forget
> about sort=_docid_ asc.
>
> On Sun, Feb 26, 2023 at 1:58 AM Fikavec F  wrote:
>
> > Thanks for the patch for testing. I could not see significant improvements
> > on virtual machines, I will try again this week on servers.
> > I tried the following values for buffers: 65536 -  64Kb, 262144 - 256Kb,
> > 524288 - 512Kb, 1048576 - 1MB, 4194304 - 4MB, 16777216 - 16MB, 33554432 -
> > 32Mb, 67108864 - 64Mb, 134217728 - 128MB. I changed the buffer size of Solr
> > and Jetty. It was visible in the logs:
> >
> > 2023-02-25 19:01:42.201 DEBUG (qtp1812823171-22) [   ]
> > o.a.s.c.u.FastWriter checking OS env for BUFSIZE =>
> > java.lang.NumberFormatException: null
> > 2023-02-25 19:01:42.201 INFO  (qtp1812823171-22) [   ]
> > o.a.s.c.u.FastWriter FastWriter.BUFSIZE=4194304
> >
> > I noticed that increasing the buffer reduces %wait on the core down to 0
> > and also with 100% loaded core, the speed sometimes increased to 520
> > megabits (I haven't seen such numbers before, but It's still far from
> > Gigabit+). Adding ident=false and/or wt=csv increases the speed a bit more
> > (+30/50 Mbit and wt=xml slow down -80 Mbit).
> >
> > What else in the data chain can be a bottleneck? OS and Network (network
> > interface and kernel tuned for 10-Gigabit, tested by iperf - ok), disk
> > (ramdisk), processor (except that 4.3 GHz core is not enough to transfer
> > data from Solr in single thread faster than 0.5 Gigabit) are not a
> > bottleneck, jetty is able to distribute a file at high speeds, with large
> > buffers I have now received by wget: 2023-02-25 21:51:37 (6.25 Gb/s),
> >  FastWriter.BUFSIZE now it's big too, what is the next possible bottleneck
> > in Solr software architecture to explore and search further?
> >
> > Thank you for your help, I hope if these are not natural algorithmic
> > limitations, we will be able to figure out and make Solr even better,
> > especially since with the advent of PCIe 5.0, NVME, DDR5 and Wi-Fi 7 speeds
> > close to 10 Gigabit are already commonplacem but many end-user needs still
> > dependent on single-threaded/core performance and do not get significant
> > benefits from new hardware speeds...
> >
> > Best Regards,
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> https://t.me/MUST_SEARCH
> A caveat: Cyrillic!

-
To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
For additional commands, e-mail: dev-h...@solr.apache.org

Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-25 Thread Mikhail Khludnev

As being said above, the speed of streaming file to socket, via OS
internals is hardly achievable with java code crunching bytes through heap.
Using RamDirectory might push on GC ,it's rather better to stick with the
default one and leave enough RAM for file cache.
Regarding  the actual params: q=*%3A*&rows=10 I'm not sure there's a
shortcut bypassing ordering results through heap (even *:* score is
constant), quite often sort=_docid_ asc allows to avoid sorting, try it.
Also, there is https://solr.apache.org/guide/8_6/exporting-result-sets.html
which might be better suites for straightforward download, don't forget
about sort=_docid_ asc.

On Sun, Feb 26, 2023 at 1:58 AM Fikavec F  wrote:

> Thanks for the patch for testing. I could not see significant improvements
> on virtual machines, I will try again this week on servers.
> I tried the following values for buffers: 65536 -  64Kb, 262144 - 256Kb,
> 524288 - 512Kb, 1048576 - 1MB, 4194304 - 4MB, 16777216 - 16MB, 33554432 -
> 32Mb, 67108864 - 64Mb, 134217728 - 128MB. I changed the buffer size of Solr
> and Jetty. It was visible in the logs:
>
> 2023-02-25 19:01:42.201 DEBUG (qtp1812823171-22) [   ]
> o.a.s.c.u.FastWriter checking OS env for BUFSIZE =>
> java.lang.NumberFormatException: null
> 2023-02-25 19:01:42.201 INFO  (qtp1812823171-22) [   ]
> o.a.s.c.u.FastWriter FastWriter.BUFSIZE=4194304
>
> I noticed that increasing the buffer reduces %wait on the core down to 0
> and also with 100% loaded core, the speed sometimes increased to 520
> megabits (I haven't seen such numbers before, but It's still far from
> Gigabit+). Adding ident=false and/or wt=csv increases the speed a bit more
> (+30/50 Mbit and wt=xml slow down -80 Mbit).
>
> What else in the data chain can be a bottleneck? OS and Network (network
> interface and kernel tuned for 10-Gigabit, tested by iperf - ok), disk
> (ramdisk), processor (except that 4.3 GHz core is not enough to transfer
> data from Solr in single thread faster than 0.5 Gigabit) are not a
> bottleneck, jetty is able to distribute a file at high speeds, with large
> buffers I have now received by wget: 2023-02-25 21:51:37 (6.25 Gb/s),
>  FastWriter.BUFSIZE now it's big too, what is the next possible bottleneck
> in Solr software architecture to explore and search further?
>
> Thank you for your help, I hope if these are not natural algorithmic
> limitations, we will be able to figure out and make Solr even better,
> especially since with the advent of PCIe 5.0, NVME, DDR5 and Wi-Fi 7 speeds
> close to 10 Gigabit are already commonplacem but many end-user needs still
> dependent on single-threaded/core performance and do not get significant
> benefits from new hardware speeds...
>
> Best Regards,
>


-- 
Sincerely yours
Mikhail Khludnev
https://t.me/MUST_SEARCH
A caveat: Cyrillic!

RE: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-25 Thread Fikavec F

Thanks for the patch for testing. I could not see significant improvements on virtual machines, I will try again this week on servers.I tried the following values for buffers: 65536 -  64Kb, 262144 - 256Kb, 524288 - 512Kb, 1048576 - 1MB, 4194304 - 4MB, 16777216 - 16MB, 33554432 - 32Mb, 67108864 - 64Mb, 134217728 - 128MB. I changed the buffer size of Solr and Jetty. It was visible in the logs:2023-02-25 19:01:42.201 DEBUG (qtp1812823171-22) [   ] o.a.s.c.u.FastWriter checking OS env for BUFSIZE => java.lang.NumberFormatException: null2023-02-25 19:01:42.201 INFO  (qtp1812823171-22) [   ] o.a.s.c.u.FastWriter FastWriter.BUFSIZE=4194304I noticed that increasing the buffer reduces %wait on the core down to 0 and also with 100% loaded core, the speed sometimes increased to 520 megabits (I haven't seen such numbers before, but It's still far from Gigabit+). Adding ident=false and/or wt=csv increases the speed a bit more (+30/50 Mbit and wt=xml slow down -80 Mbit). What else in the data chain can be a bottleneck? OS and Network (network interface and kernel tuned for 10-Gigabit, tested by iperf - ok), disk (ramdisk), processor (except that 4.3 GHz core is not enough to transfer data from Solr in single thread faster than 0.5 Gigabit) are not a bottleneck, jetty is able to distribute a file at high speeds, with large buffers I have now received by wget: 2023-02-25 21:51:37 (6.25 Gb/s),  FastWriter.BUFSIZE now it's big too, what is the next possible bottleneck in Solr software architecture to explore and search further? Thank you for your help, I hope if these are not natural algorithmic limitations, we will be able to figure out and make Solr even better, especially since with the advent of PCIe 5.0, NVME, DDR5 and Wi-Fi 7 speeds close to 10 Gigabit are already commonplacem but many end-user needs still dependent on single-threaded/core performance and do not get significant benefits from new hardware speeds... Best Regards,

RE: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-25 Thread Fikavec F

Today I created a 1GB collection with RAMDirectoryFactory and only id, stored unindexed without docValues text_sn fields in the schema. In solrconfig.xml:... and in  section:${solr.lock.type:single} In managed-schema:...        ... Receiving all data from it is also carried out at slow speeds of approximately 300 Megabits (454 maximum at two+ runs) - one 4.3 Ghz core on  VM with Solr fully load (%wait is 50%-70%), wget output at the end - 2023-02-25 12:59:27 (454 Mb/s) - ‘/dev/null’ saved [898511406]) not Gigabit+. This is 10 times slower than downloading a 10Gb file from the same Solr Jetty (as shown above - 4.83 Gigabit/sec). If you have a large collection, you can also estimate the speed of single threaded uncompressed data receiving, for example, using wget (change the ip address, collection name and rows number to your values, but so that the server response at least gigabyte. You can also use nload to track outgoing speed on the server.): wget --report-speed=bits -O /dev/null --server-response --header='Accept-Encoding: ' --post-data 'q=*%3A*&rows=10' http://192.168.220.135:8983/solr/test_collection/select Please let me know if anyone gets a speed level of at least 1Gigabit+ and even more so of the order of 5Gigabit+, which according to my tests is in the capabilities of Solr Jetty. Best Regards,
-
To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org
For additional commands, e-mail: dev-h...@solr.apache.org

Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-25 Thread Mikhail Khludnev

Hi,
I made it tunable. It should capture it from OS env BUFSIZE=8192, and java
property -DBUFSIZE=8192.
It logs actual value under INFO for FastWriter
https://github.com/mkhludnev/lucene-solr/commit/f0ed425bcbb16c50d01f3ff7fba3879148f50568
Here's a jar
https://github.com/mkhludnev/lucene-solr/releases/tag/8.11.2-SNAPSHOT,
which one can drop into the installation. You absolutely need to re/move
solr-solrj-8.11.2.jar from the release.
Happy experimenting!



On Sat, Feb 25, 2023 at 1:43 AM Fikavec F  wrote:

> jetty.xml used in default solr configuration, file downloaded by http and
> nginx maybe faster because in /etc/nginx/sites-available/default - location
> / { aio threads; sendfile on; }
> I'm reproduce this in Ubuntu on virtual machines in home PC:
>
> # 0 set /etc/sysctl.conf settings from
> https://darksideclouds.wordpress.com/2016/10/10/tuning-10gb-nics-highway-to-hell/
> # and for network adapter ens160
> sudo ip link set ens160 txqueuelen 1
> sudo ip link set ens160 mtu 9000
>
> # 1 create 16Gb ram disk
> sudo mkdir -p /mnt/solrramdisk
> sudo mount -t tmpfs -o size=16G tmpfs /mnt/solrramdisk
>
> # 2 download and start solr
> cd /mnt/solrramdisk
> wget -O solr-8.11.2.tgz
> https://www.apache.org/dyn/closer.lua/lucene/solr/8.11.2/solr-8.11.2.tgz?action=download
> tar xzf solr-8.11.2.tgz
> ./solr-8.11.2/bin/solr start
>
> # 3 create 10G test file
> head -c 10G /dev/urandom >
> /mnt/solrramdisk/solr-8.11.2/server/solr-webapp/webapp/sample.bin
>
> # on other host with same mtu and /etc/sysctl.conf settings
> # 4 download test file from Solr Jetty
> wget --report-speed=bits -O /dev/null
> http://192.168.220.135:8983/solr/sample.bin
> # Result - 2023-02-24 21:42:29 (4.83 Gb/s) - ‘/dev/null’ saved
> [10737418240]
>
> # 5 download test file from nginx (in nginx set root /mnt/solrramdisk;)
> wget --report-speed=bits -O /dev/null
> http://192.168.220.135/solr-8.11.2/server/solr-webapp/webapp/sample.bin
> # Result - 2023-02-24 21:34:40 (7.28 Gb/s) - ‘/dev/null’ saved
> [10737418240/10737418240]
>
> In virtual network with two virtual machines on the same host jetty faster
> (than 1.2 Gbit on my real hardware tests), but still slower Nginx and it
> doesn't explain why scrolling is significantly slower than Jetty's
> capabilities.
>
>Unfortunately, I am not a java developer and will not be able to offer
> a patch or rebuild the jar with a bigger buffer myself (without
> instructions on how to do it from scratch), but I can test/benchmark a new
> binary or any other Solr settings to find ways to speed up Solr Scroll.
>I'm hoping for a bigger buffer in FastWriter.java will speed up single
> thread data receiving, but this is a hypothesis that is better evaluated by
> specialists in the internal and network architecture of Solr. In my tests
> 1) Solr Jetty is able to give data much faster than they are given from
> collections. 2) Data from the collection is given at the same speed on
> servers that are quite different in performance (CPU cores are 30% slower,
> sas raid disk instead of ramdrive) with 10 Gbit network. 3) Multiple scroll
> does not increase the speed (disk and other Solr, OS caches do not help).
>Is there a mention somewhere that Solr, under any conditions, was able
> to send data from the collection faster than 400 Mbit in a single
> (single-threaded) connection? In which direction except FastWriter.java
> buffers can be searched for, what on modern fast hardware limits the data
> transfer rate during big collection scroll?
>


-- 
Sincerely yours
Mikhail Khludnev
https://t.me/MUST_SEARCH
A caveat: Cyrillic!

RE: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

2023-02-24 Thread Fikavec F

jetty.xml used in default solr configuration, file downloaded by http and nginx maybe faster because in /etc/nginx/sites-available/default - location / { aio threads; sendfile on; }I'm reproduce this in Ubuntu on virtual machines in home PC:# 0 set /etc/sysctl.conf settings from https://darksideclouds.wordpress.com/2016/10/10/tuning-10gb-nics-highway-to-hell/# and for network adapter ens160sudo ip link set ens160 txqueuelen 1sudo ip link set ens160 mtu 9000 # 1 create 16Gb ram disksudo mkdir -p /mnt/solrramdisksudo mount -t tmpfs -o size=16G tmpfs /mnt/solrramdisk # 2 download and start solr cd /mnt/solrramdiskwget -O solr-8.11.2.tgz  https://www.apache.org/dyn/closer.lua/lucene/solr/8.11.2/solr-8.11.2.tgz?action="">tar xzf solr-8.11.2.tgz ./solr-8.11.2/bin/solr start # 3 create 10G test filehead -c 10G /dev/urandom > /mnt/solrramdisk/solr-8.11.2/server/solr-webapp/webapp/sample.bin # on other host with same mtu and /etc/sysctl.conf settings # 4 download test file from Solr Jetty wget --report-speed=bits -O /dev/null http://192.168.220.135:8983/solr/sample.bin# Result - 2023-02-24 21:42:29 (4.83 Gb/s) - ‘/dev/null’ saved [10737418240] # 5 download test file from nginx (in nginx set root /mnt/solrramdisk;)wget --report-speed=bits -O /dev/null http://192.168.220.135/solr-8.11.2/server/solr-webapp/webapp/sample.bin# Result - 2023-02-24 21:34:40 (7.28 Gb/s) - ‘/dev/null’ saved [10737418240/10737418240]In virtual network with two virtual machines on the same host jetty faster (than 1.2 Gbit on my real hardware tests), but still slower Nginx and it doesn't explain why scrolling is significantly slower than Jetty's capabilities.    Unfortunately, I am not a java developer and will not be able to offer a patch or rebuild the jar with a bigger buffer myself (without instructions on how to do it from scratch), but I can test/benchmark a new binary or any other Solr settings to find ways to speed up Solr Scroll.   I'm hoping for a bigger buffer in FastWriter.java will speed up single thread data receiving, but this is a hypothesis that is better evaluated by specialists in the internal and network architecture of Solr. In my tests 1) Solr Jetty is able to give data much faster than they are given from collections. 2) Data from the collection is given at the same speed on servers that are quite different in performance (CPU cores are 30% slower, sas raid disk instead of ramdrive) with 10 Gbit network. 3) Multiple scroll does not increase the speed (disk and other Solr, OS caches do not help).   Is there a mention somewhere that Solr, under any conditions, was able to send data from the collection faster than 400 Mbit in a single (single-threaded) connection? In which direction except FastWriter.java buffers can be searched for, what on modern fast hardware limits the data transfer rate during big collection scroll?

Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: RE: Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

Re: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

RE: Re: Low untunable default FastWriter output buffer - possible reason for slow single threaded data receiving from Solr on 1Gigabit+ networks while scroll, search etc

27 matches

Site Navigation

Mail list logo

Footer information