2010/5/17 Alois Schlögl <[email protected]>: > Jaroslav Hajek wrote: >> >> On Sun, May 16, 2010 at 8:08 PM, Alois Schlögl <[email protected]> >> wrote: >>> >>> Jaroslav Hajek wrote: >>>> >>>> On Fri, May 14, 2010 at 9:15 AM, Carlo de Falco >>>> <[email protected]> >>>> wrote: >>>>> >>>>> On 13 May 2010, at 23:00, Lukas Reichlin wrote: >>>>> >>>>>>> Certainly. But at this event, only Windows machines were available >>>>>>> and >>>>>>> Matlab was already pre-installed. I had little options >>>>>>> about the setup. (and i do not want to say it loud, but M is still >>>>>>> faster, [1]. Admittingly, Octave has improved since 2006, and I see >>>>>>> only >>>>>>> difference by a factor of 2-3 instead of 4-5). >>>>>>> >>>>>>> [1] http://arxiv.org/abs/cs/0603001v1 >>>>> >>>>> Do you still have the code to run those benchmarks at hand? >>>>> It would be very interesting to compare how the speed of Octave >>>>> changed between version 2.9.4 and 3.3, >>>>> for a fair comparison you would need to run the tests on the same >>>>> hardware though.. >>>>> c. >>>>> >>>> >>>> Checking the BioSig code, I even found snippets such as: >>>> >>>> % Instantaneous heart period hp >>>> for k=2:length(t) >>>> hp(k)=(t(k)-t(k-1)); >>>> end >>>> hp(1)=NaN; >>>> hp_mean=mean(hp) >>>> >>>> and hp is not even preallocated. Of course, code like this has no >>>> chance to be efficient in Octave. >>>> Another example snippet of extremely poorly vectorized code is >>>> >>>> j=1; >>>> n_all=ones(1,length(ke_all)); %number of consecutive ectopic beats >>>> while j<length(ke_all) >>>> while ((j<=length(ke_all)-1)&((ke_all(j+1)-ke_all(j))==1)) >>>> ke_all(j)=[]; %deletion of consecutive indices >>>> n_all(j)=n_all(j)+1; >>>> end; >>>> j=j+1; >>>> end; >>>> n_all(length(ke_all)+1:end)=[]; >>>> >>>> not only there is a double loop, there is also a vector operation >>>> (element deletion) in the inner loop. >>>> I would expect, however, that this is sluggish in Matlab as well, no >>>> JIT can probably handle this. >>>> >>> >>> What is your point here ? Some people are not as good in coding as >>> others, the function this student wrote is still useful. What does it >>> say about the performance of Octave vs. M ? Zero, Nil, Nothing. >>> >> >> That was exactly the point - that code like this says nothing about >> the relative performance, because it is a slow code on both systems. >> >>> BTW, this function is not part of the biosig benchmark. >>> >> >> OK, it was a source picked randomly. If the benchmark contains code >> like this, the point still holds. There is no question that Octave is >> slower with loops than Matlab. But even if Octave performs much slower >> on a given code, often the code can be optimized so that Octave's >> speed becomes comparable to Matlab's (sometimes it even outperforms >> it). It just requires more skill than with Matlab, so especially >> people used to Matlab as their primary platform often miss good >> optimization opportunities. >> >> > > here are the benchmark results (using bench_biosig) on the same platform (a > quadcore amd64 processor). > > Version: 3.3.51+ > cputime toc[s] task > ================================ > 1.050 1.051 sload l1.gdf > 0.920 0.922 bandpower > 9.470 9.466 wackermann > 1.760 1.761 barlow > 1.800 1.799 hjorth > 0.680 0.681 TDP > 109.670 109.750 aar #1 > 109.180 109.218 aar #2 > 112.570 112.574 aar #3 > 43.160 49.353 findclassifier bp LD3 > 43.590 43.590 findclassifier barlow LD3 > 43.530 43.540 findclassifier hjorth LD3 > 43.350 43.350 findclassifier aar LD3 > 43.100 43.094 findclassifier Wackermann LD3 > 43.050 43.052 findclassifier TDP LD3 > ----------------------------------- > 606.880 613.199 total > > Version: 7.6.0.324 (R2008a) > cputime toc[s] task > ================================ > 0.770 1.272 sload l1.gdf > 1.120 1.077 bandpower > 13.190 13.156 wackermann > 1.540 1.637 barlow > 1.560 1.688 hjorth > 1.410 1.445 TDP > 42.210 42.419 aar #1 > 38.310 38.315 aar #2 > 38.650 38.577 aar #3 > 7.820 6.071 findclassifier bp LD3 > 9.000 5.990 findclassifier barlow LD3 > 8.890 5.892 findclassifier hjorth LD3 > 8.790 5.849 findclassifier aar LD3 > 8.990 5.909 findclassifier Wackermann LD3 > 8.620 5.808 findclassifier TDP LD3 > ----------------------------------- > 190.870 175.104 total
Alois, Thanks for running this benchmarks, but actually what I was interested in was comparing Octave 3.3.51 to Octave 2.9.4, not to Matlab. I thought this would be possible because you have published the results for 2.9.4 in a paper you sent a link to in a previous post and maybe you still have the machine(s) on which you ran the benchmarks at that time... > aar is basically a Kalman filter algorithm which is a recursive algorithm; > findclassifier is contains a classifier in combination with a leave-k-out > cross-validation procedure. Both problems can not be vectorized with the > standard language elements of the M-language. > > An alternative solution would be the implementation in an other language > (like C/C++) but that's beyond the point here. The user sees that Octave is > performing more than 3 times slower. From a user point of view, this is > legitimate reason to use M. > > Trying to shift all the blame to someone else (bad code, code optimized only > for matlab, etc.) is missing the point. I for sure was not blaming anyone of anything, if you got this impression you must have been mistaken. > Lets look at another area: most OpenOffice users are using it on top of > windows. And it does not hurt linux that they do so. It's similar here, > toolboxes can be used with both. Why not using octave toolboxes on top of M > ? It does no harm to octave if anyone does so. most Octave packages are licensed under GPL, so no one can take away from you the freedom of producing matlab ports or a fork that is more careful about compatibility with matlab. before you put a lot of effort into this project, though, I think you should consider that quite a few Octave-Forge packages are meant to be compatible with some matlab toolbox that already exists. Most users who have bought Matlab and care about the application fields covered by those toolboxes, would most likely have bought the toolboxes already. So I don't think you can expect a wide user base for your port. Maybe making sure that OF packages reproducing the functionality of some ML toolbox have a compatible API would be a more popular/beneficial project, I am sure that if you were to propose patches to OF for this purpose most mainatiners would gratefully accept them. > Alois > Just my 2 Cents c. ------------------------------------------------------------------------------ _______________________________________________ Octave-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/octave-dev
