NightOwl888 commented on pull request #310:
URL: https://github.com/apache/lucenenet/pull/310#issuecomment-661002167
Thanks for putting this together.
I suspect the reason you are not seeing an improvement between versions is
because you are using a `RAMDirectory`, which bypasses the codecs and some
other components that we have improved.
I added benchmarks using the `IndexFiles` and `SearchFiles` from
`Lucene.Net.Demo` as templates. These are set up to use a fixed seed to
generate the same data over and over again so we don't have to add several
megabytes of test files to the repo just to do a benchmark.
Index speed doesn't seem to have improved, but search speed has doubled
since 4.8.0-beta00005 (the one with the most downloads on NuGet.org). Here are
the results for .NET Core 3.1 on on Windows.
## Indexing
``` ini
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900
(1909/November2018Update/19H2)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6
physical cores
.NET Core SDK=3.1.301
[Host] : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT [AttachedDebugger]
4.8.0-beta00005 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
InvocationCount=1 IterationCount=15 LaunchCount=2
UnrollFactor=1 WarmupCount=10
```
| Method | Job | NuGetReferences
| Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 |
Allocated |
|----------- |---------------- |-------------------------------------------
|---------:|---------:|---------:|-----------:|----------:|----------:|----------:|
| IndexFiles | 4.8.0-beta00005 | Lucene.Net.Analysis.Common 4.8.0-beta00005
| 734.9 ms | 37.14 ms | 55.58 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.09
MB |
| IndexFiles | 4.8.0-beta00006 | Lucene.Net.Analysis.Common 4.8.0-beta00006
| 703.7 ms | 13.92 ms | 20.83 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.07
MB |
| IndexFiles | 4.8.0-beta00007 | Lucene.Net.Analysis.Common 4.8.0-beta00007
| 705.3 ms | 15.95 ms | 23.38 ms | 43000.0000 | 8000.0000 | 7000.0000 | 221
MB |
| IndexFiles | 4.8.0-beta00008 | Lucene.Net.Analysis.Common 4.8.0-beta00008
| 747.0 ms | 42.43 ms | 59.47 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.39
MB |
| IndexFiles | 4.8.0-beta00009 | Lucene.Net.Analysis.Common 4.8.0-beta00009
| 725.3 ms | 25.59 ms | 36.70 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.31
MB |
| IndexFiles | 4.8.0-beta00010 | Lucene.Net.Analysis.Common 4.8.0-beta00010
| 724.1 ms | 15.41 ms | 22.10 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.23
MB |
## Searching
``` ini
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.900
(1909/November2018Update/19H2)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6
physical cores
.NET Core SDK=3.1.301
[Host] : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT [AttachedDebugger]
4.8.0-beta00005 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.5 (CoreCLR 4.700.20.26901, CoreFX
4.700.20.27001), X64 RyuJIT
IterationCount=100 LaunchCount=3 WarmupCount=15
```
| Method | Job |
NuGetReferences | Mean | Error | StdDev |
Median | Gen 0 | Gen 1 | Gen 2 | Allocated |
|------------ |----------------
|----------------------------------------------------------------------------------
|----------:|---------:|----------:|----------:|----------:|----------:|------:|----------:|
| SearchFiles | 4.8.0-beta00005 | Lucene.Net.Analysis.Common
4.8.0-beta00005,Lucene.Net.QueryParser 4.8.0-beta00005 | 171.18 ms | 2.004 ms |
10.283 ms | 169.51 ms | 9000.0000 | 1000.0000 | - | 41.35 MB |
| SearchFiles | 4.8.0-beta00006 | Lucene.Net.Analysis.Common
4.8.0-beta00006,Lucene.Net.QueryParser 4.8.0-beta00006 | 169.86 ms | 2.666 ms |
13.754 ms | 168.14 ms | 9000.0000 | 1000.0000 | - | 41.35 MB |
| SearchFiles | 4.8.0-beta00007 | Lucene.Net.Analysis.Common
4.8.0-beta00007,Lucene.Net.QueryParser 4.8.0-beta00007 | 173.67 ms | 3.417 ms |
17.595 ms | 169.20 ms | 9000.0000 | 1000.0000 | - | 41.26 MB |
| SearchFiles | 4.8.0-beta00008 | Lucene.Net.Analysis.Common
4.8.0-beta00008,Lucene.Net.QueryParser 4.8.0-beta00008 | 95.97 ms | 1.178 ms |
6.068 ms | 95.84 ms | 8666.6667 | 500.0000 | - | 40.33 MB |
| SearchFiles | 4.8.0-beta00009 | Lucene.Net.Analysis.Common
4.8.0-beta00009,Lucene.Net.QueryParser 4.8.0-beta00009 | 93.31 ms | 0.776 ms |
3.890 ms | 93.60 ms | 8666.6667 | 500.0000 | - | 40.33 MB |
| SearchFiles | 4.8.0-beta00010 | Lucene.Net.Analysis.Common
4.8.0-beta00010,Lucene.Net.QueryParser 4.8.0-beta00010 | 85.79 ms | 0.307 ms |
1.552 ms | 85.70 ms | 8666.6667 | 500.0000 | - | 40.19 MB |
It would be good to set up a stage on Azure DevOps to run these (off by
default) so we can see the results on different operating systems and target
frameworks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
