Re: -warmup

Fri, 18 Aug 2023 07:50:26 -0700 

Hİ,
I have edited the code aligning with the Hin-Tak’s suggestion. Here is the two 
results pages, also pushed on gitlab.

Best,
Goksu
goksu.in
On 18 Aug 2023 14:02 +0300, Werner LEMBERG <w...@gnu.org>, wrote:
> > > What happens if you use, say, `-c 100000', just running the
> > > `Get_Char_Index` test? Are the percental timing differences then
> > > still that large?
> > Actually Get_Char_Index, on the three pages I have sent in the
> > prev. mail, is higher than 6% only 4 times out of 15 total. (which is
> > seem on other tests as well).
>
> Well, the thing is that IMHO the difference should be *much* smaller –
> your HTML pages show the execution of identical code on an identical
> machine, right?
>
> > about outliers, i splitted every tests into chuncks that is sized
> > 100. Made IQR calculations and calculated average time on valid
> > chunks. you can find the result in the attachment also pushed to
> > gitlab.
>
> Thanks. Hin-Tak gave additional suggestions how to possibly improve
> the removal of outliers.
>
> > also, since statistics and benchmarking are a sciences their self, i
> > am a bit struggling while approaching the problem as well as feels
> > like out of the gsoc project scope.
>
> Indeed, the focus lately shifted from a representational aspect to a
> more thorough approach how to handle benchmarks in general. You are
> done with the first part, more or less, and it looks fine. The
> latter, however, is definitely part of the GSoC project, too, and I'm
> surprised that you think this might not be so: What are benchmark
> timings good for if the returned values are completely meaningless?
>
> In most cases, a small performance optimization in FreeType might
> yield, say, an improvement of 1%. Right now, such a change would not
> be detectable at all if using the framework you are working on – it
> would be completely hidden by noise.
>
> To summarize: Benchmark comparisons only work if there is a sound
> mathematical foundation to reduce the noise. I don't ask you to
> reinvent the wheel, but please do some more internet research and
> check existing code how to tackle such problems. I'm 100% sure that
> such code already exists (for example, the Google benchmark stuff
> mentioned in a previous e-mail, scientific papers on arXiv, etc.,
> etc.) and can be easily used, adapted, and simplified for our
> purposes.
>
>
> Werner

Freetype Benchmark Results

Warning: Baseline and Benchmark have the same commit ID!

Info

InfoBaselineBenchmark
Parameters-c 550 -w 50-c 550 -w 50
Commit ID3553148135531481
Commit Date2023-08-18 02:04:38 +03002023-08-18 02:04:38 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet

* Average time for all iterations. Smaller values are better.

** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.

Total Results

TestNBaseline (µs) Benchmark (µs)Difference (%)
Load250000 178180 190899-7.1
Load_Advances (Normal)250000 161654 1594301.4
Load_Advances (Fast)250000 1047 1130-7.9
Load_Advances (Unscaled)250000 996 1006-1.0
Render250000 278410 2766440.6
Get_Glyph250000 213564 2093142.0
Get_Char_Index235000 1012 9921.9
Iterate CMap2500 873 908-4.0
New_Face2500 12862 13004-1.1
Embolden250000 229042 2261561.3
Stroke250000 956152 9556160.1
Get_BBox250000 180922 1771082.1
Get_CBox250000 211855 2114930.2
New_Face & load glyph(s)250000 29090 287781.1
TOTAL2990000 24556582452479 0.1

Results for Roboto_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load60000 2996431568-5.3
Load_Advances (Normal)60000 29942273428.7
Load_Advances (Fast)60000 2332320.6
Load_Advances (Unscaled)60000 2202200.0
Render60000 5317953842-1.2
Get_Glyph60000 3882039532-1.8
Get_Char_Index47000 197198-0.3
Iterate CMap500 185194-4.6
New_Face500 230822671.8
Embolden60000 4110941912-2.0
Stroke60000 213674213932-0.1
Get_BBox60000 17216160356.9
Get_CBox60000 3897640102-2.9
New_Face & load glyph(s)60000 556455540.2
TOTAL1416000 4715874729300.3

Results for Arial_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 3783443165-14.1
Load_Advances (Normal)47500 37650352156.5
Load_Advances (Fast)47500 202284-40.6
Load_Advances (Unscaled)47500 192198-3.1
Render47500 6076163992-5.3
Get_Glyph47500 47014468730.3
Get_Char_Index47000 1911881.6
Iterate CMap500 1641620.6
New_Face500 271226870.9
Embolden47500 49745493860.7
Stroke47500 182028185064-1.7
Get_BBox47500 4436044575-0.5
Get_CBox47500 4663646844-0.4
New_Face & load glyph(s)47500 580956961.9
TOTAL1141000 5152985243291.8

Results for TimesNewRoman_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 5399859007-9.3
Load_Advances (Normal)47500 4127043176-4.6
Load_Advances (Fast)47500 2052050.0
Load_Advances (Unscaled)47500 194195-0.5
Render47500 70520690722.1
Get_Glyph47500 54836538261.8
Get_Char_Index47000 1981951.5
Iterate CMap500 1801668.1
New_Face500 28382849-0.4
Embolden47500 59834593460.8
Stroke47500 2393062392880.0
Get_BBox47500 5187952767-1.7
Get_CBox47500 5357455530-3.7
New_Face & load glyph(s)47500 70127013-0.0
TOTAL1141000 6358446426341.1

Results for Tahoma_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 2729033068-21.2
Load_Advances (Normal)47500 2619028596-9.2
Load_Advances (Fast)47500 202205-1.5
Load_Advances (Unscaled)47500 193197-2.1
Render47500 4617446322-0.3
Get_Glyph47500 36092359830.3
Get_Char_Index47000 2302156.3
Iterate CMap500 178203-13.7
New_Face500 25522762-8.2
Embolden47500 3899039810-2.1
Stroke47500 161374162131-0.5
Get_BBox47500 3328533466-0.5
Get_CBox47500 3579236093-0.8
New_Face & load glyph(s)47500 534853160.6
TOTAL1141000 4138904243642.5

Results for Verdana_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 290932409217.2
Load_Advances (Normal)47500 26602251015.6
Load_Advances (Fast)47500 2052040.5
Load_Advances (Unscaled)47500 1971960.5
Render47500 47776434169.1
Get_Glyph47500 368023310110.1
Get_Char_Index47000 196197-0.5
Iterate CMap500 166183-10.2
New_Face500 245224400.5
Embolden47500 39363357029.3
Stroke47500 1597701552002.9
Get_BBox47500 341843026611.5
Get_CBox47500 368783292410.7
New_Face & load glyph(s)47500 535651992.9
TOTAL1141000 419040388223-7.4

Freetype Benchmark

Freetype Benchmark Results

Warning: Baseline and Benchmark have the same commit ID!

Info

InfoBaselineBenchmark
Parameters-c 550 -w 50-c 550 -w 50
Commit IDf3dfede6f3dfede6
Commit Date2023-08-18 17:42:53 +03002023-08-18 17:42:53 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet

* Average time for all iterations. Smaller values are better.

** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.

Total Results

TestNBaseline (µs) Benchmark (µs)Difference (%)
Load250000 195162 1924241.4
Load_Advances (Normal)250000 176304 1653636.2
Load_Advances (Fast)250000 1108 1206-8.8
Load_Advances (Unscaled)250000 1087 10027.8
Render250000 287896 2817382.1
Get_Glyph250000 229158 2094608.6
Get_Char_Index235000 992 9741.8
Iterate CMap2500 869 8620.7
New_Face2500 14858 1294412.9
Embolden250000 251134 22580310.1
Stroke250000 1029943 9529077.5
Get_BBox250000 194832 1756329.9
Get_CBox250000 240592 20872913.2
New_Face & load glyph(s)250000 30726 289565.8
TOTAL2990000 26546622458002 7.4

Results for Roboto_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load60000 32608299868.0
Load_Advances (Normal)60000 2551427810-9.0
Load_Advances (Fast)60000 234236-0.4
Load_Advances (Unscaled)60000 222222-0.2
Render60000 592805337410.0
Get_Glyph60000 465643866017.0
Get_Char_Index47000 189192-1.9
Iterate CMap500 162199-22.5
New_Face500 23652473-4.6
Embolden60000 42693407974.4
Stroke60000 2319272133168.0
Get_BBox60000 172141545810.2
Get_CBox60000 590793880034.3
New_Face & load glyph(s)60000 588156733.5
TOTAL1416000 523930467197-10.8

Results for Arial_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 4671850918-9.0
Load_Advances (Normal)47500 489634110216.1
Load_Advances (Fast)47500 2202037.9
Load_Advances (Unscaled)47500 2121948.5
Render47500 68496633497.5
Get_Glyph47500 566824748016.2
Get_Char_Index47000 2051916.8
Iterate CMap500 1761665.7
New_Face500 4460264840.6
Embolden47500 653324919224.7
Stroke47500 20966818130813.5
Get_BBox47500 45346437723.5
Get_CBox47500 48516460815.0
New_Face & load glyph(s)47500 625457128.7
TOTAL1141000 601248532316-11.5

Results for TimesNewRoman_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 4952253178-7.4
Load_Advances (Normal)47500 42683417592.2
Load_Advances (Fast)47500 2042030.5
Load_Advances (Unscaled)47500 1951940.5
Render47500 6927471641-3.4
Get_Glyph47500 55683541392.8
Get_Char_Index47000 195196-0.8
Iterate CMap500 1661660.6
New_Face500 281627721.6
Embolden47500 61636608351.3
Stroke47500 2434762407781.1
Get_BBox47500 54669525333.9
Get_CBox47500 56070548822.1
New_Face & load glyph(s)47500 69316942-0.2
TOTAL1141000 643522640218-0.5

Results for Tahoma_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 381732983021.9
Load_Advances (Normal)47500 343922779619.2
Load_Advances (Fast)47500 2252250.0
Load_Advances (Unscaled)47500 2141987.5
Render47500 46312458850.9
Get_Glyph47500 36125354521.9
Get_Char_Index47000 2051983.4
Iterate CMap500 19616715.0
New_Face500 277525617.7
Embolden47500 446503854213.7
Stroke47500 18529616230812.4
Get_BBox47500 35112327526.7
Get_CBox47500 424823513017.3
New_Face & load glyph(s)47500 6473538216.8
TOTAL1141000 472629416426-11.9

Results for Verdana_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load47500 2814128512-1.3
Load_Advances (Normal)47500 2475226896-8.7
Load_Advances (Fast)47500 224339-51.0
Load_Advances (Unscaled)47500 24419420.5
Render47500 4453447489-6.6
Get_Glyph47500 34104337301.1
Get_Char_Index47000 1981961.0
Iterate CMap500 1681661.5
New_Face500 24422490-2.0
Embolden47500 36822364361.0
Stroke47500 1595771551982.7
Get_BBox47500 424933111626.8
Get_CBox47500 34445338361.8
New_Face & load glyph(s)47500 51865246-1.2
TOTAL1141000 413332401844-2.8

Freetype Benchmark

Reply via email to