[PD] benchmarking slow startup with Pd-0.55 (was: Pd 0.55-0 test version released)

[IOhannes m zmoelnig]

On 5/27/24 21:58, Benjamin Wesch wrote:

thanks a lot for pointing out 'hyperfine' and for providing this
somewhat self-defeating plugin.

thanks for the additional data.

====== recap ======

i think the most promising paths for investigation are:
- the code
- the build environment
- the signing process

ad signing process:


ad build environment:

this sped up builds considerably (from about ~7 minutes to ~1 minute).




ad code:
my gut feeling tells me that there are two promising candidates:
- the changed GUI startup protocol
- something about the audio backends



====== more benchamrks ======



```
hyperfine \
        --export-csv benchmark.csv  \
        -N --warmup 3 \
        -L pd 0.54-1,0.55-0test3-4-g618b6325  \
        -L audio nosound,jack,pa  \
        -L callback ,no \
        -L quit '-send "pd quit" -nogui','-send "pd quit"','-path quitter/'  \

```

this tests:
- multiple versions of Pd
- different audio backends (w/wo callbacks)
- w/wo gui
- quiting from either GUI or core










my prelaminary findings are:

- the most important factors for the entire startup-shutdown cycle are
  1. whether there's a GUI (slow) or not (fast)

- as far as the audio backends are concerned:



my numbers differ greatly from ben's.

this can probably be explained in part by the different CPUs and OS version.


the default audio backend for macOS is PortAudio.


gfdasr
IOhannes

pd,gui,quitter,audio,callbacks,mean,stddev,max,min,median
0.55-0test3-4-g618b6325,False,core,JACK,False,71.84,4.02,83.52,67.59,70.31
0.54-1,False,core,JACK,False,73.10,1.73,81.08,70.37,73.00
0.54-1,False,core,JACK,True,73.13,1.11,76.02,70.98,73.13
0.55-0test3-4-g618b6325,False,core,JACK,True,82.42,26.50,202.31,68.34,76.39
0.55-0test3-4-g618b6325,False,core,,True,344.30,54.67,430.82,276.87,317.42
0.55-0test3-4-g618b6325,False,core,PortAudio,False,342.82,41.97,399.71,263.58,340.82
0.54-1,False,core,,False,361.04,71.35,511.08,266.48,352.23
0.54-1,False,core,PortAudio,True,349.16,54.32,422.04,254.70,360.37
0.55-0test3-4-g618b6325,False,core,,False,391.82,85.15,552.61,292.34,375.78
0.54-1,False,core,PortAudio,False,390.36,66.17,511.27,308.40,377.00
0.55-0test3-4-g618b6325,False,core,PortAudio,True,366.82,87.38,503.79,238.56,378.11
0.54-1,False,core,,True,404.86,87.37,531.86,264.42,411.97
0.55-0test3-4-g618b6325,True,core,JACK,True,861.08,75.81,992.23,790.53,823.06
0.54-1,True,core,JACK,False,899.80,94.25,1153.62,847.93,861.95
0.55-0test3-4-g618b6325,True,core,JACK,False,851.92,40.97,907.20,797.86,863.22
0.54-1,True,core,JACK,True,887.35,89.99,1065.12,723.93,866.73
0.55-0test3-4-g618b6325,True,core,PortAudio,False,1144.84,78.64,1310.04,1076.81,1104.88
0.55-0test3-4-g618b6325,True,core,,True,1711.05,1676.74,6476.86,1092.16,1149.70
0.54-1,True,core,,False,1193.95,129.82,1466.68,1046.32,1157.56
0.55-0test3-4-g618b6325,True,core,PortAudio,True,1147.98,58.61,1222.08,1053.13,1158.35
0.54-1,True,core,,True,1197.02,50.36,1297.91,1127.83,1181.75
0.55-0test3-4-g618b6325,True,core,,False,1215.56,118.42,1416.62,1081.07,1189.53
0.54-1,True,core,PortAudio,False,1205.03,95.07,1359.36,1070.87,1193.81
0.54-1,True,core,PortAudio,True,1228.20,106.38,1421.10,1077.11,1215.69
0.54-1,True,GUI,JACK,True,1401.18,69.36,1573.92,1322.67,1389.49
0.54-1,True,GUI,JACK,False,1945.65,1608.85,6518.35,1324.70,1434.31
0.55-0test3-4-g618b6325,True,GUI,JACK,False,1581.72,85.13,1786.99,1479.25,1551.14
0.55-0test3-4-g618b6325,True,GUI,JACK,True,1564.22,52.01,1663.76,1503.96,1555.87
0.54-1,True,GUI,,True,1737.18,156.37,2035.41,1617.41,1670.74
0.54-1,True,GUI,PortAudio,False,1740.17,134.53,2031.50,1629.80,1682.92
0.54-1,True,GUI,PortAudio,True,1695.70,92.82,1822.31,1510.91,1709.85
0.54-1,True,GUI,,False,2245.71,1636.79,6898.83,1668.10,1712.10
0.55-0test3-4-g618b6325,True,GUI,PortAudio,True,1916.14,102.95,2073.05,1814.75,1873.23
0.55-0test3-4-g618b6325,True,GUI,,True,1896.39,109.35,2147.04,1782.45,1878.90
0.55-0test3-4-g618b6325,True,GUI,PortAudio,False,2965.98,2236.20,7275.45,1779.43,1883.85
0.55-0test3-4-g618b6325,True,GUI,,False,1918.37,170.22,2245.15,1703.66,1963.89

pd,gui,quitter,audio,callbacks,mean,stddev,max,min,median
0.55-0test3-4-g618b6325,False,core,OSS,False,1.56,0.37,4.87,1.04,1.51
0.54.1+ds-5,False,core,OSS,False,1.55,0.31,4.87,1.07,1.51
0.55-0test3-4-g618b6325,False,core,JACK,False,1.50,0.13,2.86,1.07,1.51
0.55-0test3-4-g618b6325,False,core,JACK,True,1.74,0.72,5.42,1.11,1.51
0.54.1+ds-5,False,core,JACK,True,1.58,0.42,5.20,1.13,1.51
0.54.1+ds-5,False,core,JACK,False,1.56,0.32,4.75,1.02,1.52
0.55-0test3-4-g618b6325,False,core,OSS,True,1.55,0.29,4.61,1.07,1.52
0.54.1+ds-5,False,core,OSS,True,1.72,0.66,5.43,1.06,1.54
0.54.1+ds-5,False,core,ALSA,False,2.16,0.73,6.80,1.43,2.01
0.54.1+ds-5,False,core,ALSA,True,2.03,0.37,6.25,1.41,2.01
0.54.1+ds-5,False,core,,False,2.13,0.60,6.60,1.43,2.02
0.55-0test3-4-g618b6325,False,core,ALSA,False,2.09,0.50,6.66,1.39,2.02
0.54.1+ds-5,False,core,,True,2.01,0.19,5.04,1.54,2.02
0.55-0test3-4-g618b6325,False,core,,False,2.05,0.45,6.83,1.41,2.03
0.55-0test3-4-g618b6325,False,core,ALSA,True,2.12,0.64,6.77,1.41,2.03
0.55-0test3-4-g618b6325,False,core,,True,2.16,0.70,6.66,1.44,2.04
0.55-0test3-4-g618b6325,True,core,JACK,False,39.98,4.26,59.49,34.01,40.07
0.55-0test3-4-g618b6325,True,core,OSS,False,41.40,5.34,61.72,34.25,40.08
0.55-0test3-4-g618b6325,True,core,OSS,True,40.49,4.10,54.56,33.19,40.20
0.55-0test3-4-g618b6325,True,core,JACK,True,41.56,5.43,61.03,34.38,40.72
0.55-0test3-4-g618b6325,True,core,ALSA,True,41.50,5.24,59.59,34.05,40.80
0.55-0test3-4-g618b6325,True,core,ALSA,False,40.90,4.52,51.50,34.37,40.99
0.55-0test3-4-g618b6325,True,core,,True,42.44,4.67,57.19,34.66,41.69
0.55-0test3-4-g618b6325,True,core,,False,43.80,5.88,63.69,35.48,42.54
0.55-0test3-4-g618b6325,False,core,PortAudio,True,59.03,5.28,68.58,50.44,59.42
0.54.1+ds-5,False,core,PortAudio,True,60.48,5.50,72.76,51.32,60.04
0.54.1+ds-5,False,core,PortAudio,False,60.57,5.73,72.87,50.97,60.13
0.55-0test3-4-g618b6325,False,core,PortAudio,False,62.65,6.76,81.18,50.61,63.25
0.55-0test3-4-g618b6325,True,core,PortAudio,False,101.48,14.13,135.18,83.63,95.24
0.55-0test3-4-g618b6325,True,core,PortAudio,True,99.99,6.79,113.85,90.61,99.58
0.54.1+ds-5,True,core,ALSA,False,125.21,6.89,140.05,114.55,125.57
0.54.1+ds-5,True,core,OSS,False,135.25,16.86,184.33,115.77,129.36
0.54.1+ds-5,True,core,JACK,False,130.91,11.57,155.63,116.98,129.71
0.54.1+ds-5,True,core,ALSA,True,133.67,11.44,159.22,118.06,133.59
0.54.1+ds-5,True,core,JACK,True,135.03,13.02,157.52,118.26,133.74
0.54.1+ds-5,True,core,OSS,True,133.36,12.74,158.49,114.25,133.99
0.54.1+ds-5,True,core,,True,135.61,12.68,159.05,118.25,135.45
0.54.1+ds-5,True,core,,False,137.89,13.66,161.86,113.93,136.12
0.55-0test3-4-g618b6325,True,GUI,OSS,False,172.72,5.78,181.28,162.97,172.66
0.55-0test3-4-g618b6325,True,GUI,,True,173.20,9.81,192.91,156.85,173.58
0.55-0test3-4-g618b6325,True,GUI,,False,175.34,12.07,197.42,161.97,173.95
0.55-0test3-4-g618b6325,True,GUI,ALSA,True,173.32,8.64,190.04,157.11,174.34
0.55-0test3-4-g618b6325,True,GUI,ALSA,False,174.86,8.85,194.50,156.51,174.80
0.54.1+ds-5,True,core,PortAudio,False,181.29,10.28,197.02,167.35,177.30
0.55-0test3-4-g618b6325,True,GUI,OSS,True,177.05,12.55,196.15,155.86,178.13
0.54.1+ds-5,True,core,PortAudio,True,185.32,8.94,203.92,169.63,183.85
0.55-0test3-4-g618b6325,True,GUI,JACK,False,186.03,14.76,214.99,169.95,191.78
0.55-0test3-4-g618b6325,True,GUI,JACK,True,191.00,12.70,213.36,169.98,192.52
0.54.1+ds-5,True,GUI,JACK,False,227.30,12.92,265.15,213.98,225.49
0.55-0test3-4-g618b6325,True,GUI,PortAudio,True,229.62,10.13,242.22,211.49,226.55
0.55-0test3-4-g618b6325,True,GUI,PortAudio,False,228.59,9.90,244.92,213.09,227.08
0.54.1+ds-5,True,GUI,,True,235.24,15.09,261.91,213.59,231.36
0.54.1+ds-5,True,GUI,JACK,True,233.63,9.87,255.48,222.26,231.83
0.54.1+ds-5,True,GUI,OSS,True,232.17,10.19,246.71,210.15,233.97
0.54.1+ds-5,True,GUI,ALSA,False,233.65,10.42,249.52,219.40,234.05
0.54.1+ds-5,True,GUI,,False,237.10,11.31,260.87,222.49,235.63
0.54.1+ds-5,True,GUI,ALSA,True,242.12,8.56,257.72,230.94,241.10
0.54.1+ds-5,True,GUI,OSS,False,239.01,10.09,250.81,219.78,242.84
0.54.1+ds-5,True,GUI,PortAudio,False,298.22,15.69,328.05,273.28,295.29
0.54.1+ds-5,True,GUI,PortAudio,True,297.04,9.93,313.59,276.82,296.79

macOS spends some time during startup to check whether the application can be trusted. in theory this should give some performance penalty at the very first startup of an application, but from then on things should be pretty fast (as the results are hashed). in any case, i'm not aware that anything has changed in the way we sign/notarize Pd since 0.54 (apart from Apple having switched to a new signing/notarizing infrastructure,...) about two months ago i switched the CI builders for macOS from an aging Mac Mini 5,2 (mid 2011) running Big Sur (within a VM on a Debian hypervisor), to a Mac Studio 13,1 M1 (2022) running Monterey (within a VM on a Ventura hypervisor). (sidenote: this change only affects the build process itself; signing/notarization is still done on the old VMs) i *guess* that the changed GUI startup will only affect Pd if it is started via the core (as opposed to starting via the GUI, e.g. by clicking on the app icon). i would have assumed that the changed GUI startup would actually speed up things (at least for our benchmark tests, where we start via the core and try to quit asap) in the meantime i've learned some more about hyperfine (which can do argument variations on its own), and ran some more tests with something like the following: "Pd-{pd}.app/Contents/Resources/bin/pd -noprefs -nostdpath -{audio} -{callback}callback {quit}" (it assumes that there are a couple of Pd-*.app in the current directory, so we only need vary the actual version number. also the quitter-plugin is in the quitter/ directory of the cwd and is *explicitly* activated). i did one run on our Mac Studio (Ventura) via VNC(!), and another on my Debian desktop PC (obviously with slightly different invocation to cater for the changed path layout and different available backends). no check has been made to see if the audio backends are actually useable (if Pd tries to open a blocked audio device, this could stall the startup considerably!) find the results in the attached CSV files (where i've turned the startup options in separate columns. an empty value in the "audio" column means that Pd was started with "-nosound") - the build environment is not to blame for the slow down (the startup times range between 70ms and almost 2 seconds (on macOS) for both the old and the new binaries (on Debian it's between 2ms and 300ms)) 2. whether the shutdown is initiated from the core (fast) or from the GUI (slow) 1. whether one uses callbacks or not, does not seem to make much of a difference 2. using PortAudio adds a considerable overhead. (JACK timings and the PortAudio timings are clearly in two different leagues, separated by a factor of 1.5 up to 5). the separation is most prominent in the tests that do not involve the GUI (suggesting that the GUI overhead hides the overhead from the audio backend) esp. when running Pd-0.54 with GUI it takes much longer (e.g. using the quit-plugin, the minimum turnaround time is ~1.4secs!, rather than 0.4secs as seen by ben) without a GUI i'm seeing comparable numbers (~70ms), but only when using JACK.

OpenPGP_signature.asc
Description: OpenPGP digital signature

_______________________________________________
Pd-list@lists.iem.at mailing list
UNSUBSCRIBE and account-management -> 
https://lists.puredata.info/listinfo/pd-list