AddictedCS commented on issue #933:
URL: https://github.com/apache/lucenenet/issues/933#issuecomment-2016568457
Hey guys,
I've reran the benchmark multiple times now and here are the results:
```
BenchmarkDotNet v0.13.12, macOS Sonoma 14.4 (23E214) [Darwin 23.4.0]
Apple M2 Max, 1 CPU, 12 logical and 12 physical cores
.NET SDK 8.0.203
[Host] : .NET 7.0.5 (7.0.523.17405), Arm64 RyuJIT AdvSIMD
.NET 7.0 : .NET 7.0.5 (7.0.523.17405), Arm64 RyuJIT AdvSIMD
.NET 8.0 : .NET 8.0.3 (8.0.324.11423), Arm64 RyuJIT AdvSIMD
Server=True
```
| Method | Job | Runtime | N | Mean | Error
| StdDev | Ratio | RatioSD |
|------------------------ |--------- |--------- |---
|------------:|----------:|-----------:|------:|--------:|
| IndexDocumentsBenchmark | .NET 7.0 | .NET 7.0 | 25 | 59.25 ms | 0.839
ms | 0.862 ms | 1.00 | 0.00 |
| IndexDocumentsBenchmark | .NET 8.0 | .NET 8.0 | 25 | 2,174.27 ms | 47.201
ms | 139.172 ms | 33.67 | 1.25 |
```
BenchmarkDotNet v0.13.12, macOS Sonoma 14.4 (23E214) [Darwin 23.4.0]
Apple M2 Max, 1 CPU, 12 logical and 12 physical cores
.NET SDK 8.0.203
[Host] : .NET 8.0.3 (8.0.324.11423), Arm64 RyuJIT AdvSIMD
.NET 7.0 : .NET 7.0.5 (7.0.523.17405), Arm64 RyuJIT AdvSIMD
.NET 8.0 : .NET 8.0.3 (8.0.324.11423), Arm64 RyuJIT AdvSIMD
Server=True
```
| Method | Job | Runtime | N | Mean | Error
| StdDev | Ratio | RatioSD |
|------------------------ |--------- |--------- |---
|------------:|----------:|-----------:|------:|--------:|
| IndexDocumentsBenchmark | .NET 7.0 | .NET 7.0 | 25 | 62.84 ms | 1.423
ms | 3.823 ms | 1.00 | 0.00 |
| IndexDocumentsBenchmark | .NET 8.0 | .NET 8.0 | 25 | 2,161.11 ms | 46.276
ms | 136.446 ms | 34.70 | 2.80 |
I can't get the 12k ratio now as I'm running it at home, and I concur with
@paulirwin that it is "only" 30x, which is still unacceptable.
@jeme thanks a deeper benchmark, here are my results
```
BenchmarkDotNet v0.13.12, macOS Sonoma 14.4 (23E214) [Darwin 23.4.0]
Apple M2 Max, 1 CPU, 12 logical and 12 physical cores
.NET SDK 8.0.203
[Host] : .NET 8.0.3 (8.0.324.11423), Arm64 RyuJIT AdvSIMD
.NET 7.0 : .NET 7.0.5 (7.0.523.17405), Arm64 RyuJIT AdvSIMD
.NET 8.0 : .NET 8.0.3 (8.0.324.11423), Arm64 RyuJIT AdvSIMD
Server=True
```
| Method | Job | Runtime | Mean | Error | StdDev
| Median | Ratio | RatioSD |
|------------ |--------- |---------
|-------------:|-----------:|-----------:|-------------:|-------:|--------:|
| Write1Kb | .NET 7.0 | .NET 7.0 | 28.18 μs | 0.337 μs | 0.299 μs
| 28.17 μs | 1.00 | 0.00 |
| Write1Kb | .NET 8.0 | .NET 8.0 | 4,675.38 μs | 89.870 μs | 150.152 μs
| 4,667.45 μs | 164.77 | 6.64 |
| | | | | |
| | | |
| Write2Kb | .NET 7.0 | .NET 7.0 | 28.12 μs | 0.413 μs | 0.366 μs
| 28.12 μs | 1.00 | 0.00 |
| Write2Kb | .NET 8.0 | .NET 8.0 | 4,633.83 μs | 90.353 μs | 253.360 μs
| 4,624.14 μs | 171.17 | 7.64 |
| | | | | |
| | | |
| Write4Kb | .NET 7.0 | .NET 7.0 | 29.03 μs | 0.240 μs | 0.201 μs
| 28.98 μs | 1.00 | 0.00 |
| Write4Kb | .NET 8.0 | .NET 8.0 | 4,449.07 μs | 88.012 μs | 179.785 μs
| 4,459.38 μs | 151.98 | 6.81 |
| | | | | |
| | | |
| Write512Kb | .NET 7.0 | .NET 7.0 | 4,815.22 μs | 94.760 μs | 141.833 μs
| 4,813.58 μs | 1.00 | 0.00 |
| Write512Kb | .NET 8.0 | .NET 8.0 | 4,997.23 μs | 97.949 μs | 120.290 μs
| 5,009.46 μs | 1.03 | 0.04 |
| | | | | |
| | | |
| Write1024Kb | .NET 7.0 | .NET 7.0 | 5,093.80 μs | 91.040 μs | 85.159 μs
| 5,106.26 μs | 1.00 | 0.00 |
| Write1024Kb | .NET 8.0 | .NET 8.0 | 5,299.68 μs | 104.049 μs | 92.237 μs
| 5,292.41 μs | 1.04 | 0.02 |
| | | | | |
| | | |
| Write16Mb | .NET 7.0 | .NET 7.0 | 12,569.03 μs | 248.962 μs | 572.032 μs
| 12,473.78 μs | 1.00 | 0.00 |
| Write16Mb | .NET 8.0 | .NET 8.0 | 13,267.28 μs | 264.196 μs | 714.270 μs
| 13,017.40 μs | 1.06 | 0.06 |
The results corroborate with @jeme, which is 150x performance degradation.
@eladmarg I respectfully disagree that this is a design bug. The pattern
that is used in the initial benchmark is quite often used by those who use
Lucene, calling it _soft commit_. This paradigm is used in Solr and other
custom implementations. The index is not flushed to the disk, by refreshing the
searcher after index writer update the changes are reflected in new searches
which is important when near-realtime updates are required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
