Re: -warmup

Wed, 16 Aug 2023 13:33:38 -0700 

Hi,
I have added the total table that you suggested.

I think Get_Char_Index is not the problem, the results varies all the time. 
Here are the three results that i had in the same minute (one has different 
flags).

Should I proceed to detect outliers?

Since we do not get the same error rate consistently,  I think we will not find 
the target we expected by outliers.

Best,
Goksu
goksu.in
On 7 Aug 2023 15:57 +0300, Werner LEMBERG <w...@gnu.org>, wrote:
>
> > > What exactly means 'Baseline (ms)'? Is the shown number the time
> > >  for one loop? For all loops together? Please clarify and mention
> > >  this on the HTML page.
> >
> > Clarified that the times are milliseconds for the cumulative time
> > for all iterations.
>
> Thanks. The sentence is not easily comprehensible. Perhaps change it
> to something like
>
> ```
> Cumulative time for all iterations. Smaller values means better.
> ```
>
> BTW, in column 'N' I see stuff like '68160 | 65880'. What does this
> mean? Please add an explanatory comment to the HTML page.
>
> Another thing: Please mention on the HTML page the completion time for
> each test, and the total execution time of all tests together.
>
> > > Looking at the 'Load_Advances (Unscaled)' row, I think that 100%
> > >  difference between 0.001 and 0.002 doesn't make any sense. How do
> > >  you compute the percentage? Is this based on the cumulative time
> > > of  all loops? If so, and you really get such small numbers, there
> > > must  be some fine-tuning for high-speed tests (for example,
> > > increasing N  for this particular test by a factor of 10, say) to
> > > get meaningful  timing values.
> >
> > it was cumulative time in milliseconds but converted it microseconds
> > as how it was and it seem got better.
>
> We are getting nearer, again :-)
>
> What worries me, though, is that we still have such enormous
> differences. For `Get_Char_Index` I think it's lack of precision.
> Please try to fix this – if the ratio
>
> cumulative_time / N
>
> is smaller than a given threshold, N must be increased a lot. In
> other words, for `Roboto_subset.ttf`, N should be set to, say, 10*N.
>
> For the other large differences I think we need some statistical
> analysis to get better results – simple cumulation is not good enough.
> In particular, outliers should be removed (at least this is my
> hypothesis). Maybe you can look up the internet to find some simple
> code to handle them.
>
> An idea to identify outliers could be to split the cumulation time
> into, say, 100 smaller intervals. You can the discard the too-large
> values and compute the mean of the remaining data. My reasoning is
> that other CPU activity happens in parallel, but only for short
> amounts of time.
>
> Have you actually done a statistical analysis of, say, 'Load_Advances
> (Normal)' for `Arial_subset.ttf`? For example, printing all timings
> of the datapoints as histograms for runs A and B? *Are* there
> outliers? Maybe there is another statistical mean value that gives
> more meaningful results.
>
>
> Werner

Freetype Benchmark Results

Warning: Baseline and Benchmark have the same commit ID!

Info

InfoBaselineBenchmark
Parameters-c 1000 -w 50-c 1000 -w 50
Commit ID4bcd97114bcd9711
Commit Date2023-08-07 15:11:28 +03002023-08-07 15:11:28 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet

* Cumulative time for all iterations. Smaller values are better.

** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.

Total Results

TestNBaseline (µs) Benchmark (µs)Difference (%)
Load500000 3482359 3519568-1.1
Load_Advances (Normal)500000 2922257 3129046-7.1
Load_Advances (Fast)500000 13671 15627-14.3
Load_Advances (Unscaled)500000 12459 15309-22.9
Render500000 1827123 1878593-2.8
Get_Glyph500000 768199 7543151.8
Get_Char_Index470000 13396 127994.5
Iterate CMap5000 9860 93784.9
New_Face5000 248768 2405913.3
Embolden500000 1107786 1109611-0.2
Stroke294205 | 289365 7804180 77762470.4
Get_BBox500000 491174 496942-1.2
Get_CBox500000 355009 355822-0.2
New_Face & load glyph(s)500000 571172 5470084.2
TOTAL5774205 | 5769365 1962741319860856 -1.2

Results for Roboto_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load120000 6119135767105.8
Load_Advances (Normal)120000 498485538696-8.1
Load_Advances (Fast)120000 29644976-67.9
Load_Advances (Unscaled)120000 27905476-96.3
Render120000 419155437405-4.4
Get_Glyph120000 1738201661314.4
Get_Char_Index94000 270126412.2
Iterate CMap1000 2084186110.7
New_Face1000 4206642190-0.3
Embolden120000 220170222545-1.1
Stroke62880 | 61080 163185216240320.5
Get_BBox120000 1081191078920.2
Get_CBox120000 8159783621-2.5
New_Face & load glyph(s)120000 104217107909-3.5
TOTAL2715960 390193339220850.5

Results for Arial_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 8533098365162.0
Load_Advances (Normal)95000 680709732802-7.7
Load_Advances (Fast)95000 3067244820.2
Load_Advances (Unscaled)95000 2527225610.7
Render95000 3577913444293.7
Get_Glyph95000 143432147740-3.0
Get_Char_Index94000 27132881-6.2
Iterate CMap1000 18261829-0.2
New_Face1000 4777650312-5.3
Embolden95000 199248213090-6.9
Stroke57665 | 56430 148541114812960.3
Get_BBox95000 95626946631.0
Get_CBox95000 6872069159-0.6
New_Face & load glyph(s)95000 103873105626-1.7
TOTAL2206095 404602840850471.0

Results for TimesNewRoman_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 916144926065-1.1
Load_Advances (Normal)95000 813784875632-7.6
Load_Advances (Fast)95000 24172424-0.3
Load_Advances (Unscaled)95000 226922580.5
Render95000 378067383493-1.4
Get_Glyph95000 1586901478196.9
Get_Char_Index94000 3060241821.0
Iterate CMap1000 2331190718.2
New_Face1000 646015439115.8
Embolden95000 2847732650596.9
Stroke44365 | 44460 15442481545269-0.1
Get_BBox95000 109597115606-5.5
Get_CBox95000 7054071610-1.5
New_Face & load glyph(s)95000 130336134454-3.2
TOTAL2180825 448085745284051.1

Results for Tahoma_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 568933628873-10.5
Load_Advances (Normal)95000 481014513665-6.8
Load_Advances (Fast)95000 24772870-15.9
Load_Advances (Unscaled)95000 22732423-6.6
Render95000 330628358159-8.3
Get_Glyph95000 143448146595-2.2
Get_Char_Index94000 245824261.3
Iterate CMap1000 18181909-5.0
New_Face1000 49076464685.3
Embolden95000 206672209651-1.4
Stroke62700 | 61845 156047915474800.8
Get_BBox95000 8971989857-0.2
Get_CBox95000 6536465460-0.1
New_Face & load glyph(s)95000 102598987873.7
TOTAL2216545 360695737146233.0

Results for Verdana_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 532060551404-3.6
Load_Advances (Normal)95000 448265468251-4.5
Load_Advances (Fast)95000 27462909-5.9
Load_Advances (Unscaled)95000 26002896-11.4
Render95000 341482355107-4.0
Get_Glyph95000 1488091460301.9
Get_Char_Index94000 246424331.3
Iterate CMap1000 18011872-3.9
New_Face1000 4524947230-4.4
Embolden95000 196923199266-1.2
Stroke66595 | 65550 158219015781700.3
Get_BBox95000 8811388924-0.9
Get_CBox95000 68788659724.1
New_Face & load glyph(s)95000 13014810023223.0
TOTAL2224145 359163836106960.5

Freetype Benchmark

Freetype Benchmark Results

Warning: Baseline and Benchmark have the same commit ID!

Info

InfoBaselineBenchmark
Parameters-c 550 -w 50-c 550 -w 50
Commit ID4bcd97114bcd9711
Commit Date2023-08-07 15:11:28 +03002023-08-07 15:11:28 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet

* Cumulative time for all iterations. Smaller values are better.

** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.

Total Results

TestNBaseline (µs) Benchmark (µs)Difference (%)
Load275000 1940945 1960259-1.0
Load_Advances (Normal)275000 1693643 1757029-3.7
Load_Advances (Fast)275000 6966 68471.7
Load_Advances (Unscaled)275000 6382 63220.9
Render275000 968631 1047055-8.1
Get_Glyph275000 410428 445338-8.5
Get_Char_Index258500 6847 7183-4.9
Iterate CMap2750 5289 51642.4
New_Face2750 137845 1372680.4
Embolden275000 584074 5811100.5
Stroke262730 | 262890 7008914 7043463-0.5
Get_BBox275000 260539 272939-4.8
Get_CBox275000 188954 210858-11.6
New_Face & load glyph(s)275000 297956 304196-2.1
TOTAL3276730 | 3276890 1351741313785031 -2.0

Results for Roboto_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load66000 3138892994864.6
Load_Advances (Normal)66000 287800296824-3.1
Load_Advances (Fast)66000 15561615-3.8
Load_Advances (Unscaled)66000 14631486-1.6
Render66000 223554243002-8.7
Get_Glyph66000 92555118479-28.0
Get_Char_Index51700 13681536-12.3
Iterate CMap550 9791078-10.1
New_Face550 2278125196-10.6
Embolden66000 1177911164941.1
Stroke62280 | 61680 162831316204860.5
Get_BBox66000 5578760158-7.8
Get_CBox66000 4288156697-32.2
New_Face & load glyph(s)66000 6467667592-4.5
TOTAL1549560 285539329101291.9

Results for Arial_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load52250 504744534834-6.0
Load_Advances (Normal)52250 351531416852-18.6
Load_Advances (Fast)52250 12671300-2.6
Load_Advances (Unscaled)52250 11971203-0.5
Render52250 183606192968-5.1
Get_Glyph52250 79899790171.1
Get_Char_Index51700 16531664-0.7
Iterate CMap550 104110410.0
New_Face550 376372978520.9
Embolden52250 1086371040534.2
Stroke52250 13405441359642-1.4
Get_BBox52250 49978495880.8
Get_CBox52250 3629136601-0.9
New_Face & load glyph(s)52250 56463551802.3
TOTAL1255100 275448828637284.0

Results for TimesNewRoman_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load52250 528080557600-5.6
Load_Advances (Normal)52250 4962444591307.5
Load_Advances (Fast)52250 1523135810.8
Load_Advances (Unscaled)52250 131312306.3
Render52250 208907217202-4.0
Get_Glyph52250 79182776821.9
Get_Char_Index51700 127312720.1
Iterate CMap550 9819641.7
New_Face550 29084267618.0
Embolden52250 139582139743-0.1
Stroke43700 | 44460 15395451540181-0.0
Get_BBox52250 5772967301-16.6
Get_CBox52250 3704346742-26.2
New_Face & load glyph(s)52250 6764671096-5.1
TOTAL1238760 318813232082620.6

Results for Tahoma_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load52250 3184273076733.4
Load_Advances (Normal)52250 270798277004-2.3
Load_Advances (Fast)52250 12741277-0.2
Load_Advances (Unscaled)52250 121611872.4
Render52250 175500197229-12.4
Get_Glyph52250 8060184321-4.6
Get_Char_Index51700 12691435-13.1
Iterate CMap550 1303104020.2
New_Face550 2435627740-13.9
Embolden52250 111778113705-1.7
Stroke52250 12508291295714-3.6
Get_BBox52250 4772849956-4.7
Get_CBox52250 3545536577-3.2
New_Face & load glyph(s)52250 5241957650-10.0
TOTAL1255100 237295324525083.4

Results for Verdana_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load52250 2758052606665.5
Load_Advances (Normal)52250 287270307219-6.9
Load_Advances (Fast)52250 134612973.6
Load_Advances (Unscaled)52250 11931216-1.9
Render52250 177064196654-11.1
Get_Glyph52250 7819185839-9.8
Get_Char_Index51700 128412760.6
Iterate CMap550 9851041-5.7
New_Face550 2398727786-15.8
Embolden52250 106286107115-0.8
Stroke52250 124968312274401.8
Get_BBox52250 49317459366.9
Get_CBox52250 37284342418.2
New_Face & load glyph(s)52250 56752526787.2
TOTAL1255100 234644723504040.2

Freetype Benchmark

Freetype Benchmark Results

Warning: Baseline and Benchmark have the same commit ID!

Info

InfoBaselineBenchmark
Parameters-c 1000 -w 50-c 1000 -w 50
Commit ID4bcd97114bcd9711
Commit Date2023-08-07 15:11:28 +03002023-08-07 15:11:28 +0300
BranchGSoC-2023-AhmetGSoC-2023-Ahmet

* Cumulative time for all iterations. Smaller values are better.

** N count in (x | y) format is for showing baseline and benchmark N counts seperately when they differs.

Total Results

TestNBaseline (µs) Benchmark (µs)Difference (%)
Load500000 3574511 33847525.3
Load_Advances (Normal)500000 3001429 3115941-3.8
Load_Advances (Fast)500000 12871 13708-6.5
Load_Advances (Unscaled)500000 11982 12662-5.7
Render500000 1829127 1895816-3.6
Get_Glyph500000 755948 7476541.1
Get_Char_Index470000 12153 12350-1.6
Iterate CMap5000 9339 91452.1
New_Face5000 233723 235328-0.7
Embolden500000 1094561 1096360-0.2
Stroke292330 | 294035 7764649 7786202-0.3
Get_BBox500000 493151 493549-0.1
Get_CBox500000 365376 3565502.4
New_Face & load glyph(s)500000 566729 5421194.3
TOTAL5772330 | 5774035 1972554919702136 0.1

Results for Roboto_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load120000 5873285730382.4
Load_Advances (Normal)120000 494478515420-4.2
Load_Advances (Fast)120000 31133278-5.3
Load_Advances (Unscaled)120000 27933210-14.9
Render120000 439328452423-3.0
Get_Glyph120000 168057168854-0.5
Get_Char_Index94000 24122629-9.0
Iterate CMap1000 18211862-2.3
New_Face1000 4123141691-1.1
Embolden120000 221434221765-0.1
Stroke63000 | 62520 162983316253800.3
Get_BBox120000 106660107050-0.4
Get_CBox120000 89839824108.3
New_Face & load glyph(s)120000 12774010432318.3
TOTAL2717520 39160673903333-0.3

Results for Arial_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 8636608120126.0
Load_Advances (Normal)95000 703405741526-5.4
Load_Advances (Fast)95000 24492520-2.9
Load_Advances (Unscaled)95000 22922388-4.2
Render95000 3493303468350.7
Get_Glyph95000 1511981450184.1
Get_Char_Index94000 24212424-0.1
Iterate CMap1000 188418133.8
New_Face1000 51398490084.6
Embolden95000 2007961972971.7
Stroke56620 | 57665 14809131485980-0.3
Get_BBox95000 95990940582.0
Get_CBox95000 6981069850-0.1
New_Face & load glyph(s)95000 1107181051755.0
TOTAL2206285 40862644055904-0.7

Results for TimesNewRoman_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 9520519171503.7
Load_Advances (Normal)95000 827119873521-5.6
Load_Advances (Fast)95000 245424251.2
Load_Advances (Unscaled)95000 228122740.3
Render95000 374973380720-1.5
Get_Glyph95000 145054146668-1.1
Get_Char_Index94000 24222440-0.7
Iterate CMap1000 18511881-1.6
New_Face1000 5096151140-0.4
Embolden95000 264007272286-3.1
Stroke43985 | 44175 15380281543158-0.3
Get_BBox95000 110972111756-0.7
Get_CBox95000 72576706142.7
New_Face & load glyph(s)95000 1319441303981.2
TOTAL2180160 447669345064310.7

Results for Tahoma_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 6022435447909.5
Load_Advances (Normal)95000 493275498016-1.0
Load_Advances (Fast)95000 24192456-1.5
Load_Advances (Unscaled)95000 23452460-4.9
Render95000 320830361343-12.6
Get_Glyph95000 1440441433260.5
Get_Char_Index94000 24132414-0.0
Iterate CMap1000 195518356.1
New_Face1000 45833457640.2
Embolden95000 206447206567-0.1
Stroke63270 | 64505 15577711567078-0.6
Get_BBox95000 8987090831-1.1
Get_CBox95000 6522267532-3.5
New_Face & load glyph(s)95000 99549101768-2.2
TOTAL2219775 363421636361800.1

Results for Verdana_subset.ttf

TestN * Baseline (µs) * Benchmark (µs) Difference (%)
Load95000 5692295377625.5
Load_Advances (Normal)95000 483152487458-0.9
Load_Advances (Fast)95000 24363029-24.3
Load_Advances (Unscaled)95000 22712330-2.6
Render95000 344666354495-2.9
Get_Glyph95000 1475951437882.6
Get_Char_Index94000 248524431.7
Iterate CMap1000 182817544.0
New_Face1000 4430047725-7.7
Embolden95000 2018771984451.7
Stroke65455 | 65170 15581041564606-0.4
Get_BBox95000 8965989854-0.2
Get_CBox95000 67929661442.6
New_Face & load glyph(s)95000 96778100455-3.8
TOTAL2222625 36123093600288-0.3

Freetype Benchmark

Reply via email to